[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:11 dist-test-slave-srgv krb5kdc[14645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:11 dist-test-slave-srgv krb5kdc[14645](info): set up 2 sockets
Aug 10 23:06:11 dist-test-slave-srgv krb5kdc[14645](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:06:12.376323 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:12 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:12.513329 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:33617
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:33617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:12.627152 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:12.627499 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:12.627593 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:12.631800 14658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:12.631902 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:12.631939 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:12.631979 14658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:12.632007 14658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:12.637960 14658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:33617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:12.639170 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:12.640264 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:12.648931 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:12.649008 14658 server_base.cc:1034] running on GCE node
W20260810 23:06:12.648942 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:12.648940 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:12.649910 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:12.651141 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:12.652338 14658 hybrid_clock.cc:648] HybridClock initialized: now 1786403172652281 us; error 74 us; skew 500 ppm
Aug 10 23:06:12 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:12.662204 14658 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:12.663363 14658 webserver.cc:460] Webserver started at http://127.14.75.62:45621/ using document root <none> and password file <none>
I20260810 23:06:12.664001 14658 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:12.664122 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:12.664366 14658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:12.666778 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "caec8abbfdc442fbb05b39795d55b451"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "df3da79a4b399c7fd6ec61fb4cf2dccc"
server_key_iv: "7283784c77603f30980438165876c4fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:12.667445 14658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "caec8abbfdc442fbb05b39795d55b451"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "df3da79a4b399c7fd6ec61fb4cf2dccc"
server_key_iv: "7283784c77603f30980438165876c4fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:12.672008 14658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:12.674755 14673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:12.676251 14658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:12.676420 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "caec8abbfdc442fbb05b39795d55b451"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "df3da79a4b399c7fd6ec61fb4cf2dccc"
server_key_iv: "7283784c77603f30980438165876c4fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:12.676555 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:12.723372 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:12.727228 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:12.727464 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:12.735390 14658 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:33617
I20260810 23:06:12.735410 14727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:33617 every 8 connection(s)
I20260810 23:06:12.736413 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:12.739238 14728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:12.741659 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 14658
I20260810 23:06:12.741835 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:12.743106 14636 external_mini_cluster.cc:1428] Setting key f5178db06113b655fcc64bd166d8f6e6
I20260810 23:06:12.745144 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451: Bootstrap starting.
I20260810 23:06:12.748021 14728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:12.748965 14728 log.cc:824] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:12.751348 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451: No bootstrap required, opened a new log
I20260810 23:06:12.756024 14728 raft_consensus.cc:348] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } }
I20260810 23:06:12.756250 14728 raft_consensus.cc:374] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:12.756310 14728 raft_consensus.cc:729] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caec8abbfdc442fbb05b39795d55b451, State: Initialized, Role: FOLLOWER
I20260810 23:06:12.756733 14728 consensus_queue.cc:260] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } }
I20260810 23:06:12.756860 14728 raft_consensus.cc:388] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:12.756950 14728 raft_consensus.cc:482] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:12.757052 14728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:12.758302 14728 raft_consensus.cc:504] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } }
I20260810 23:06:12.758685 14728 leader_election.cc:302] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caec8abbfdc442fbb05b39795d55b451; no voters:
I20260810 23:06:12.759023 14728 leader_election.cc:288] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:12.759172 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:12.759410 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [term 1 LEADER]: Becoming Leader. State: Replica: caec8abbfdc442fbb05b39795d55b451, State: Running, Role: LEADER
I20260810 23:06:12.759765 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } }
I20260810 23:06:12.760320 14728 sys_catalog.cc:564] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:12.761962 14737 sys_catalog.cc:455] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caec8abbfdc442fbb05b39795d55b451. Latest consensus state: current_term: 1 leader_uuid: "caec8abbfdc442fbb05b39795d55b451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } } }
I20260810 23:06:12.762127 14737 sys_catalog.cc:458] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:12.761958 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caec8abbfdc442fbb05b39795d55b451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caec8abbfdc442fbb05b39795d55b451" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33617 } } }
I20260810 23:06:12.762384 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:12.762626 14741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 23:06:12 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:12.765965 14741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:12.771047 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:12.746269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.14.75.62:33617)
0810 23:06:12.746795 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:12.746815 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:12.747858 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 23:06:12.751976 (+ 4118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:12.751991 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:12.752305 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:12.752569 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:12.755703 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:12.756862 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:12.758645 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:12.759297 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:12.760926 (+ 1629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:12.764907 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:12.764917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:12.764936 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:12.764977 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:12.766890 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:12.767826 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:12.767834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:12.767839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:12.768025 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:12.768434 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:12.768438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:12.768440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:12.768912 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:12.769149 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:12.769783 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 23:06:12.770067 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":148,"threads_started":1}
I20260810 23:06:12.773547 14741 catalog_manager.cc:1347] Generated new cluster ID: 1df0bb5ad52a4b6b90855294625df58a
I20260810 23:06:12.773614 14741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:12.786821 14741 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:12.788151 14741 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:12.804157 14741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caec8abbfdc442fbb05b39795d55b451: Generated new TSK 0
I20260810 23:06:12.805092 14741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:12.840000 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:12.960515 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:12.960831 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:12.960918 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:12.965538 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:12.965624 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:12.965741 14759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:12.971414 14759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:12.972618 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:12.973747 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:12.984062 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:12.984069 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:12.984069 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:12.984114 14759 server_base.cc:1034] running on GCE node
I20260810 23:06:12.984867 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:12.985623 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:12.986826 14759 hybrid_clock.cc:648] HybridClock initialized: now 1786403172986813 us; error 32 us; skew 500 ppm
Aug 10 23:06:12 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:12.989825 14759 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:12.990741 14759 webserver.cc:460] Webserver started at http://127.14.75.1:46799/ using document root <none> and password file <none>
I20260810 23:06:12.991286 14759 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:12.991348 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:12.991521 14759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:12.993681 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "600dfa6debd348bea430edc84aa7e6d5"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "e3cbae646cbe140254997bfa2bd85ffd"
server_key_iv: "8111214537f69382390439917a9e74cc"
server_key_version: "encryptionkey@0"
I20260810 23:06:12.994305 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "600dfa6debd348bea430edc84aa7e6d5"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "e3cbae646cbe140254997bfa2bd85ffd"
server_key_iv: "8111214537f69382390439917a9e74cc"
server_key_version: "encryptionkey@0"
I20260810 23:06:12.998148 14759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:13.000751 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.002208 14759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:06:13.002343 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "600dfa6debd348bea430edc84aa7e6d5"
format_stamp: "Formatted at 2026-08-10 23:06:12 on dist-test-slave-srgv"
server_key: "e3cbae646cbe140254997bfa2bd85ffd"
server_key_iv: "8111214537f69382390439917a9e74cc"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.002468 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:13.029259 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:13.032231 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:13.032451 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:13.033087 14759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:13.034039 14759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:13.034088 14759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.034158 14759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:13.034189 14759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.043883 14759 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:41561
I20260810 23:06:13.043907 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:41561 every 8 connection(s)
I20260810 23:06:13.044938 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:13.047549 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 14759
I20260810 23:06:13.047691 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:13.047976 14636 external_mini_cluster.cc:1428] Setting key c9e1844e46943e287eb351d001f275d7
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:13.060587 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.046893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:43689 (local address 127.14.75.62:33617)
0810 23:06:13.047051 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:13.047056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:13.048398 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 23:06:13.049380 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:13.049382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:13.049476 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:13.049565 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:13.050785 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.051578 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.053161 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.053617 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.053757 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.056613 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:13.056616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:13.056618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:13.056645 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:13.058284 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.058836 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.058839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.058840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.058920 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.059284 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.059288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.059290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.059495 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:13.059601 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:13.060175 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 23:06:13.060373 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:13.061533 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.047409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33617 (local address 127.14.75.1:43689)
0810 23:06:13.048247 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:13.048298 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:13.049144 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:13.049746 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:13.049753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:13.050126 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:13.050623 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.050636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.051721 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.051738 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.053004 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.053015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.053790 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.053793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.053943 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.054623 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:13.054641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:13.056386 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:13.058446 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.058451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.058465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.058728 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.059020 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.059022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.059023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.059163 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.059615 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:13.059624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:13.059914 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 23:06:13.060124 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 23:06:13.060505 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":134,"threads_started":1}
I20260810 23:06:13.063036 14892 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33617
I20260810 23:06:13.063357 14892 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:13.064046 14892 heartbeater.cc:511] Master 127.14.75.62:33617 requested a full tablet report, sending...
I20260810 23:06:13.065874 14691 ts_manager.cc:194] Registered new tserver with Master: 600dfa6debd348bea430edc84aa7e6d5 (127.14.75.1:41561)
I20260810 23:06:13.067049 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:43689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:13.078888 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:13.202486 14897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:13.202795 14897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:13.202843 14897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:13.207186 14897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:13.207257 14897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:13.207322 14897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:13.212844 14897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:13.213984 14897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:13.214980 14897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:13.224938 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:13.225162 14897 server_base.cc:1034] running on GCE node
W20260810 23:06:13.224939 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:13.224952 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:13.225867 14897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:13.226532 14897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:13.227702 14897 hybrid_clock.cc:648] HybridClock initialized: now 1786403173227688 us; error 32 us; skew 500 ppm
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:13.230744 14897 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:13.231711 14897 webserver.cc:460] Webserver started at http://127.14.75.2:43295/ using document root <none> and password file <none>
I20260810 23:06:13.232297 14897 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:13.232362 14897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:13.232611 14897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:13.234885 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "6a1e40193b344d8d968e8a7c26268247"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f06440316b5b599df3e510ebab95694"
server_key_iv: "d9fa25a542e3defc5ba6d1a013dcfe7a"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.235548 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "6a1e40193b344d8d968e8a7c26268247"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f06440316b5b599df3e510ebab95694"
server_key_iv: "d9fa25a542e3defc5ba6d1a013dcfe7a"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.239881 14897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:13.242501 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.243796 14897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:13.243943 14897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "6a1e40193b344d8d968e8a7c26268247"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f06440316b5b599df3e510ebab95694"
server_key_iv: "d9fa25a542e3defc5ba6d1a013dcfe7a"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.244089 14897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:13.265712 14897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:13.268869 14897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:13.269083 14897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:13.269757 14897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:13.270648 14897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:13.270718 14897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.270790 14897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:13.270843 14897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.280638 14897 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37823
I20260810 23:06:13.280675 15030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37823 every 8 connection(s)
I20260810 23:06:13.281849 14897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:13.286830 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 14897
I20260810 23:06:13.286963 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:13.287199 14636 external_mini_cluster.cc:1428] Setting key 752c6e293c9f9fb3f5147b2490937cbe
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:13.297385 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.283788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:57643 (local address 127.14.75.62:33617)
0810 23:06:13.283939 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:13.283942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:13.285021 (+ 1079us) server_negotiation.cc:407] Connection header received
0810 23:06:13.286001 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:13.286005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:13.286063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:13.286173 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:13.287448 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.288152 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.289894 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.290115 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.290201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.293010 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:13.293017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:13.293024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:13.293061 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:13.294981 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.295544 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.295550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.295553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.295656 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.296076 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.296081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.296083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.296364 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:13.296477 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:13.296996 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:06:13.297139 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:13.298192 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.284190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33617 (local address 127.14.75.2:57643)
0810 23:06:13.284871 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:13.284913 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:13.285796 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:13.286352 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:13.286359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:13.286744 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:13.287296 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.287317 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.288283 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.288286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.289765 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.289780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.290283 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.290289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.290520 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.291265 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:13.291282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:13.292805 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:13.295118 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.295129 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.295144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.295428 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.295754 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.295756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.295758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.295970 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.296474 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:13.296479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:13.296742 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 23:06:13.296903 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:06:13.297179 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":120,"threads_started":1}
I20260810 23:06:13.299433 15031 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33617
I20260810 23:06:13.299649 15031 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:13.300209 15031 heartbeater.cc:511] Master 127.14.75.62:33617 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:06:13.301469 14691 ts_manager.cc:194] Registered new tserver with Master: 6a1e40193b344d8d968e8a7c26268247 (127.14.75.2:37823)
I20260810 23:06:13.302060 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:57643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:13.316334 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:13.433118 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:13.433485 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:13.433597 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:13.437867 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:13.437970 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:13.438067 15036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:13.443774 15036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33617
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:13.444991 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:13.446125 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:13.456519 15043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:13.456561 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:13.456522 15041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:13.457018 15036 server_base.cc:1034] running on GCE node
I20260810 23:06:13.457495 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:13.458227 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:13.459416 15036 hybrid_clock.cc:648] HybridClock initialized: now 1786403173459397 us; error 27 us; skew 500 ppm
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:13.462800 15036 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:13.463841 15036 webserver.cc:460] Webserver started at http://127.14.75.3:38125/ using document root <none> and password file <none>
I20260810 23:06:13.464468 15036 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:13.464536 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:13.464722 15036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:13.467121 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "5ee76572c9ff4c5f9f2d4d9c30e890fd"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f6a02675d5ba649910a5af5694774b7"
server_key_iv: "66b381aaf81075af1e0bac1b9acfa03c"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.467800 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee76572c9ff4c5f9f2d4d9c30e890fd"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f6a02675d5ba649910a5af5694774b7"
server_key_iv: "66b381aaf81075af1e0bac1b9acfa03c"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.472659 15036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:06:13.475454 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.476799 15036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:06:13.476962 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "5ee76572c9ff4c5f9f2d4d9c30e890fd"
format_stamp: "Formatted at 2026-08-10 23:06:13 on dist-test-slave-srgv"
server_key: "5f6a02675d5ba649910a5af5694774b7"
server_key_iv: "66b381aaf81075af1e0bac1b9acfa03c"
server_key_version: "encryptionkey@0"
I20260810 23:06:13.477129 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:13.533836 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:13.537485 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:13.537714 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:13.538354 15036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:13.539216 15036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:13.539290 15036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.539363 15036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:13.539407 15036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:13.549737 15036 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:39731
I20260810 23:06:13.549774 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:39731 every 8 connection(s)
I20260810 23:06:13.550747 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:13.554071 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15036
I20260810 23:06:13.554175 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:13.554426 14636 external_mini_cluster.cc:1428] Setting key 7540284d77718c63bb2070df436d5e9d
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:13.565330 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.552550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:60309 (local address 127.14.75.62:33617)
0810 23:06:13.552722 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:13.552727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:13.553855 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 23:06:13.554849 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:13.554853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:13.554915 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:13.555005 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:13.556511 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.557169 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.558419 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.558629 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.558710 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.561392 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:13.561395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:13.561397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:13.561440 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:13.563099 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.563771 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.563774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.563775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.563861 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.564307 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.564310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.564312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.564528 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:13.564616 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:13.565105 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 23:06:13.565208 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:06:13.566205 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.552937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33617 (local address 127.14.75.3:60309)
0810 23:06:13.553676 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:13.553720 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:13.554633 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:13.555211 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:13.555217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:13.555811 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:13.556377 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.556391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.557317 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.557320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.558305 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:13.558315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.558758 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.558760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:13.558909 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.559594 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:13.559610 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:13.561216 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:13.563235 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.563241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.563288 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.563650 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.563954 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:13.563956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:13.563957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:13.564196 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:13.564625 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:13.564630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:13.564875 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 23:06:13.565036 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:06:13.565311 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":126,"threads_started":1}
I20260810 23:06:13.567459 15169 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33617
I20260810 23:06:13.567723 15169 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:13.568291 15169 heartbeater.cc:511] Master 127.14.75.62:33617 requested a full tablet report, sending...
I20260810 23:06:13.569391 14691 ts_manager.cc:194] Registered new tserver with Master: 5ee76572c9ff4c5f9f2d4d9c30e890fd (127.14.75.3:39731)
I20260810 23:06:13.570021 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:60309
I20260810 23:06:13.581686 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:13.613059 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.603663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.14.75.1:41561)
0810 23:06:13.603933 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:13.603940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:13.603966 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:06:13.604060 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:13.604065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:13.604258 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:13.604383 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:13.605114 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.605978 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.606901 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.607317 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.607403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.609888 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:13.609899 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:13.609913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:13.609939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:13.611445 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.611857 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.611862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.611863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.611941 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.612190 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.612192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.612193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.612427 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:13.612576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:13.612782 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:06:13.612906 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
W20260810 23:06:13.614113 14804 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34770
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[14645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:13.641659 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:13.632887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.14.75.1:41561)
0810 23:06:13.633066 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:13.633071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:13.633190 (+ 119us) server_negotiation.cc:407] Connection header received
0810 23:06:13.633374 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:13.633378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:13.633472 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:13.633567 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:13.634298 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.634817 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.635671 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:13.635913 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:13.636025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:13.638540 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:13.638544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:13.638547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:13.638582 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:13.639890 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.640320 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.640324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.640326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.640411 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:13.640700 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:13.640705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:13.640707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:13.640949 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:13.641092 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:13.641343 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 23:06:13.641489 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:06:13.642750 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 14759
I20260810 23:06:13.650180 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 14897
I20260810 23:06:13.656608 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15036
I20260810 23:06:13.663431 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 14658
2026-08-10T23:06:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2394 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[15184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[15184](info): set up 2 sockets
Aug 10 23:06:13 dist-test-slave-srgv krb5kdc[15184](info): commencing operation
krb5kdc: starting...
W20260810 23:06:14.717880 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:14 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:14.843878 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46729
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:14.956369 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:14.956647 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:14.956687 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:14.960911 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:14.960976 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:14.960996 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:14.961014 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:14.961069 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:14.966566 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:14.967648 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:14.968654 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:14.977366 15207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:14.977347 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:14.977347 15205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:14.977908 15198 server_base.cc:1034] running on GCE node
I20260810 23:06:14.978454 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:14.979563 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:14.980746 15198 hybrid_clock.cc:648] HybridClock initialized: now 1786403174980736 us; error 29 us; skew 500 ppm
Aug 10 23:06:14 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403174, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:14.983489 15198 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:14.984393 15198 webserver.cc:460] Webserver started at http://127.14.75.62:35349/ using document root <none> and password file <none>
I20260810 23:06:14.984973 15198 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:14.985066 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:14.985293 15198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:14.987535 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "76953a02a3ae4d50aeccc28248a4df31"
format_stamp: "Formatted at 2026-08-10 23:06:14 on dist-test-slave-srgv"
server_key: "fa4473fcee5c85b6e7cbaa82193f0132"
server_key_iv: "c18e8bc1ff5946767a19bdfb8b11c69a"
server_key_version: "encryptionkey@0"
I20260810 23:06:14.988131 15198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "76953a02a3ae4d50aeccc28248a4df31"
format_stamp: "Formatted at 2026-08-10 23:06:14 on dist-test-slave-srgv"
server_key: "fa4473fcee5c85b6e7cbaa82193f0132"
server_key_iv: "c18e8bc1ff5946767a19bdfb8b11c69a"
server_key_version: "encryptionkey@0"
I20260810 23:06:14.992019 15198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:14.994431 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:14.995594 15198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:14.995735 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "76953a02a3ae4d50aeccc28248a4df31"
format_stamp: "Formatted at 2026-08-10 23:06:14 on dist-test-slave-srgv"
server_key: "fa4473fcee5c85b6e7cbaa82193f0132"
server_key_iv: "c18e8bc1ff5946767a19bdfb8b11c69a"
server_key_version: "encryptionkey@0"
I20260810 23:06:14.995854 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:15.030583 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:15.033533 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:15.033747 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:15.040820 15198 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46729
I20260810 23:06:15.040850 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46729 every 8 connection(s)
I20260810 23:06:15.041821 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:15.044505 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:15.050032 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31: Bootstrap starting.
I20260810 23:06:15.050928 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15198
I20260810 23:06:15.051045 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:15.051349 14636 external_mini_cluster.cc:1428] Setting key d06e59d6c476af9ccde180a833152b18
I20260810 23:06:15.053049 15267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:15.053843 15267 log.cc:824] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:15.056118 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31: No bootstrap required, opened a new log
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:15.060847 15267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } }
I20260810 23:06:15.061128 15267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:15.061200 15267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76953a02a3ae4d50aeccc28248a4df31, State: Initialized, Role: FOLLOWER
I20260810 23:06:15.061720 15267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } }
I20260810 23:06:15.061910 15267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:15.061982 15267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:15.062076 15267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:15.063380 15267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } }
I20260810 23:06:15.063777 15267 leader_election.cc:302] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76953a02a3ae4d50aeccc28248a4df31; no voters:
I20260810 23:06:15.064152 15267 leader_election.cc:288] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:15.064234 15272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:15.064429 15272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [term 1 LEADER]: Becoming Leader. State: Replica: 76953a02a3ae4d50aeccc28248a4df31, State: Running, Role: LEADER
I20260810 23:06:15.064723 15272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } }
I20260810 23:06:15.065532 15267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:15.066382 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76953a02a3ae4d50aeccc28248a4df31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } } }
I20260810 23:06:15.066548 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:15.066859 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76953a02a3ae4d50aeccc28248a4df31. Latest consensus state: current_term: 1 leader_uuid: "76953a02a3ae4d50aeccc28248a4df31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76953a02a3ae4d50aeccc28248a4df31" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46729 } } }
I20260810 23:06:15.066964 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:15.067953 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.052588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.14.75.62:46729)
0810 23:06:15.053031 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.053046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.053120 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:06:15.053853 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.053870 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.054285 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.054703 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:15.055597 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.056613 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.057781 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.058254 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.058372 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.061561 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:15.061567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:15.061584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:15.061616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:15.063843 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.064423 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.064432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.064438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.064610 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.064943 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.064946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.064948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.065359 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:15.065711 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.066614 (+ 903us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.066929 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":151,"threads_started":1}
I20260810 23:06:15.067981 15281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:15.071072 15281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:15.076471 15281 catalog_manager.cc:1347] Generated new cluster ID: 4cd7db029e064ae182933c291a09ed6a
I20260810 23:06:15.076551 15281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:15.103595 15281 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:15.104624 15281 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:15.112651 15281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76953a02a3ae4d50aeccc28248a4df31: Generated new TSK 0
I20260810 23:06:15.113451 15281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:15.148329 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:15.263171 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:15.263551 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:15.263629 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:15.268114 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:15.268185 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:15.268292 15294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:15.273708 15294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:15.274861 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:15.275885 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:15.285533 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:15.285531 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:15.285533 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:15.286369 15294 server_base.cc:1034] running on GCE node
I20260810 23:06:15.286772 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:15.287446 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:15.288628 15294 hybrid_clock.cc:648] HybridClock initialized: now 1786403175288606 us; error 41 us; skew 500 ppm
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:15.291603 15294 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:15.292480 15294 webserver.cc:460] Webserver started at http://127.14.75.1:43529/ using document root <none> and password file <none>
I20260810 23:06:15.293006 15294 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:15.293076 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:15.293251 15294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:15.295351 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "7d0e500d623c450bb0bd7644c5eef5c8"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "4af2984f5aa59e85e29dddbba214e564"
server_key_iv: "85607533a503b517f170b4397ec990c2"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.295876 15294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0e500d623c450bb0bd7644c5eef5c8"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "4af2984f5aa59e85e29dddbba214e564"
server_key_iv: "85607533a503b517f170b4397ec990c2"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.299609 15294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:15.302068 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.303254 15294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:15.303352 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "7d0e500d623c450bb0bd7644c5eef5c8"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "4af2984f5aa59e85e29dddbba214e564"
server_key_iv: "85607533a503b517f170b4397ec990c2"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.303442 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:15.325278 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:15.328608 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:15.328827 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:15.329478 15294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:15.330394 15294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:15.330440 15294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.330504 15294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:15.330535 15294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.339430 15294 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:39677
I20260810 23:06:15.339509 15428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:39677 every 8 connection(s)
I20260810 23:06:15.340431 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:15.345499 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15294
I20260810 23:06:15.345606 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:15.345865 14636 external_mini_cluster.cc:1428] Setting key 60d8b265708fb4afc8b7f791883ecf4e
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:15.355269 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.342299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:35163 (local address 127.14.75.62:46729)
0810 23:06:15.342435 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.342438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.343473 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 23:06:15.344386 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.344390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.344453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.344555 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:15.345850 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.346512 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.348052 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.348286 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.348370 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.351083 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:15.351090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:15.351096 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:15.351132 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:15.352958 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.353531 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.353538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.353541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.353652 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.354011 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.354014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.354017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.354276 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:15.354394 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.354915 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.355071 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:06:15.356048 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.342601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46729 (local address 127.14.75.1:35163)
0810 23:06:15.343305 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.343346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.344173 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.344729 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.344737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.345106 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:15.345694 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.345710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.346665 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.346667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.347917 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.347931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.348447 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.348454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.348692 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.349324 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:15.349340 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:15.350891 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:15.353094 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.353100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.353114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.353386 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.353760 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.353762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.353764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.353905 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.354381 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:15.354386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:15.354639 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.354803 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.355095 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:15.357290 15429 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46729
I20260810 23:06:15.357589 15429 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:15.358109 15429 heartbeater.cc:511] Master 127.14.75.62:46729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:15.359902 15231 ts_manager.cc:194] Registered new tserver with Master: 7d0e500d623c450bb0bd7644c5eef5c8 (127.14.75.1:39677)
I20260810 23:06:15.361478 15231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:35163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:15.375815 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:15.490527 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:15.490857 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:15.490937 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:15.495111 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:15.495217 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:15.495311 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:15.500952 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:15.502162 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:15.503289 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:15.513475 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:15.513584 15434 server_base.cc:1034] running on GCE node
W20260810 23:06:15.513437 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:15.513478 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:15.514106 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:15.514770 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:15.515944 15434 hybrid_clock.cc:648] HybridClock initialized: now 1786403175515918 us; error 40 us; skew 500 ppm
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:15.518867 15434 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:15.519721 15434 webserver.cc:460] Webserver started at http://127.14.75.2:38881/ using document root <none> and password file <none>
I20260810 23:06:15.520259 15434 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:15.520321 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:15.520509 15434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:15.522670 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "c5478222305b429b99f898090d83fe5e"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "b0c984c15152bbe4e9365e758a732066"
server_key_iv: "61a53061e729bbf958bd1f525d2c5d59"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.523221 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "c5478222305b429b99f898090d83fe5e"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "b0c984c15152bbe4e9365e758a732066"
server_key_iv: "61a53061e729bbf958bd1f525d2c5d59"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.527235 15434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:15.529659 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.530869 15434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:15.531001 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "c5478222305b429b99f898090d83fe5e"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "b0c984c15152bbe4e9365e758a732066"
server_key_iv: "61a53061e729bbf958bd1f525d2c5d59"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.531114 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:15.541785 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:15.545162 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:15.545380 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:15.546027 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:15.546861 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:15.546936 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.547017 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:15.547062 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.556634 15434 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:42689
I20260810 23:06:15.556653 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:42689 every 8 connection(s)
I20260810 23:06:15.557577 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:15.562621 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15434
I20260810 23:06:15.562767 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:15.563095 14636 external_mini_cluster.cc:1428] Setting key 9ae3aeeb7b7891cec31c745fa0590a4c
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:15.572826 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.559505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:50925 (local address 127.14.75.62:46729)
0810 23:06:15.559645 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.559649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.560754 (+ 1105us) server_negotiation.cc:407] Connection header received
0810 23:06:15.561676 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.561678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.561725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.561801 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:15.563075 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.563758 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.565311 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.565606 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.565719 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.568710 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:15.568717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:15.568723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:15.568758 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:15.570584 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.571103 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.571110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.571114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.571218 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.571565 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.571570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.571575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.571860 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:15.571987 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.572480 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.572623 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:15.573639 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.559795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46729 (local address 127.14.75.2:50925)
0810 23:06:15.560540 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.560606 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.561493 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.561951 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.561958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.562354 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:15.562921 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.562937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.563929 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.563932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.565187 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.565201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.565763 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.565766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.565943 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.566643 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:15.566670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:15.568525 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:15.570709 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.570714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.570728 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.570994 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.571318 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.571320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.571321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.571458 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.571971 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:15.571976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:15.572229 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.572391 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.572672 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:15.575049 15571 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46729
I20260810 23:06:15.575305 15571 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:15.575852 15571 heartbeater.cc:511] Master 127.14.75.62:46729 requested a full tablet report, sending...
I20260810 23:06:15.577329 15231 ts_manager.cc:194] Registered new tserver with Master: c5478222305b429b99f898090d83fe5e (127.14.75.2:42689)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:06:15.578110 15231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:50925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:15.594194 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:15.707442 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:15.707715 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:15.707762 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:15.711890 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:15.711946 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:15.712006 15576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:15.717535 15576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:15.718575 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:15.719549 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:15.729302 15582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:15.729322 15584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:15.729343 15581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:15.729637 15576 server_base.cc:1034] running on GCE node
I20260810 23:06:15.730099 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:15.730729 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:15.731881 15576 hybrid_clock.cc:648] HybridClock initialized: now 1786403175731868 us; error 28 us; skew 500 ppm
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:15.734750 15576 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:15.735742 15576 webserver.cc:460] Webserver started at http://127.14.75.3:33371/ using document root <none> and password file <none>
I20260810 23:06:15.736306 15576 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:15.736397 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:15.736634 15576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:15.738725 15576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "b291f61f2ee9431d8086055980a9061d"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "1c6aec525095af367044fe834e24f153"
server_key_iv: "9e6e89f8a78523740328cacf430d2aeb"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.739295 15576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "b291f61f2ee9431d8086055980a9061d"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "1c6aec525095af367044fe834e24f153"
server_key_iv: "9e6e89f8a78523740328cacf430d2aeb"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.743144 15576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:15.745606 15591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.746824 15576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:15.746958 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "b291f61f2ee9431d8086055980a9061d"
format_stamp: "Formatted at 2026-08-10 23:06:15 on dist-test-slave-srgv"
server_key: "1c6aec525095af367044fe834e24f153"
server_key_iv: "9e6e89f8a78523740328cacf430d2aeb"
server_key_version: "encryptionkey@0"
I20260810 23:06:15.747069 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:15.759465 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:15.762578 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:15.762799 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:15.763609 15576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:15.764623 15576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:15.764707 15576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.764780 15576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:15.764829 15576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:15.774782 15576 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:37943
I20260810 23:06:15.774801 15713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:37943 every 8 connection(s)
I20260810 23:06:15.775722 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:15.781163 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15576
I20260810 23:06:15.781296 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:15.781633 14636 external_mini_cluster.cc:1428] Setting key 3640c6787abf851c5a6ed4a9640edb79
Aug 10 23:06:15 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:15.789283 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.777512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:43527 (local address 127.14.75.62:46729)
0810 23:06:15.777681 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.777686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.778580 (+ 894us) server_negotiation.cc:407] Connection header received
0810 23:06:15.779416 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.779419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.779465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.779540 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:15.780673 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.781404 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.782811 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.783026 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.783145 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.785774 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:15.785777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:15.785779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:15.785805 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:15.787185 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.787732 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.787740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.787741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.787819 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.788169 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.788174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.788175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.788364 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:15.788470 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.788989 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.789119 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:06:15.790172 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.777956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46729 (local address 127.14.75.3:43527)
0810 23:06:15.778411 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.778454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.779220 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.779702 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.779712 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.780067 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:15.780536 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.780548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.781556 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.781559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.782690 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.782701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.783176 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.783179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.783325 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.783978 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:15.783995 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:15.785591 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:15.787320 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.787326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.787344 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.787621 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.787910 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:15.787914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:15.787917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:15.788074 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:15.788469 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:15.788478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:15.788747 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.788945 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.789302 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260810 23:06:15.791285 15714 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46729
I20260810 23:06:15.791548 15714 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:15.792060 15714 heartbeater.cc:511] Master 127.14.75.62:46729 requested a full tablet report, sending...
I20260810 23:06:15.793099 15231 ts_manager.cc:194] Registered new tserver with Master: b291f61f2ee9431d8086055980a9061d (127.14.75.3:37943)
I20260810 23:06:15.793740 15231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:43527
I20260810 23:06:15.795620 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:15.806174 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.799058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.14.75.62:46729)
0810 23:06:15.799208 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.799211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.799427 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:06:15.799527 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.799529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.799567 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.799636 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:15.800384 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.800887 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.801777 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.801995 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.802075 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.803420 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:15.803423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:15.803425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:15.803441 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:15.804653 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.805026 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.805028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.805029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.805099 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:15.805332 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:15.805334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:15.805335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:15.805596 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:15.805704 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.805905 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.806002 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:15.810901 15231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:15.813875 15231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:15.828220 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.822369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:37943 (local address 127.0.0.1:56250)
0810 23:06:15.823175 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.823229 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.823363 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.824123 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.824131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.824158 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:15.824300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.824308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.825393 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.825399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.826510 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.826521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.827654 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.827657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.827771 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.827807 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.827922 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.828056 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260810 23:06:15.828224 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.822609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.14.75.3:37943)
0810 23:06:15.823373 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.823388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.823416 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:06:15.823501 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.823506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.823726 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.823894 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:15.824440 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.825254 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.826686 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.827528 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.827652 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.827809 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.827958 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.828055 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":120,"threads_started":1}
I20260810 23:06:15.828751 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.822769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39677 (local address 127.0.0.1:59992)
0810 23:06:15.823560 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.823569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.823631 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.824395 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.824398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.824428 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:15.824701 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.824706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.826134 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.826137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.827291 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.827302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.828366 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.828370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.828467 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.828489 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.828539 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.828619 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":211,"spinlock_wait_cycles":9472,"thread_start_us":94,"threads_started":1}
I20260810 23:06:15.829046 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.822574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:42689 (local address 127.0.0.1:60464)
0810 23:06:15.823283 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.823295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.823363 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.824395 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.824398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.824424 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:15.824540 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.824547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.826289 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.826293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.827285 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.827295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.828366 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.828370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.828699 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.828730 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.828811 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.828897 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":225,"spinlock_wait_cycles":384,"thread_start_us":120,"threads_started":1}
I20260810 23:06:15.829457 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.823556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.14.75.2:42689)
0810 23:06:15.823925 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.823937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.823959 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:06:15.824052 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.824057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.824229 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.824883 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:15.825049 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.826153 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.827389 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.828217 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.828415 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.828555 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.829006 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.829268 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":133,"threads_started":1}
I20260810 23:06:15.829483 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.823193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.14.75.1:39677)
0810 23:06:15.823979 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.823993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.824012 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:15.824092 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.824096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.824245 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.824488 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:15.824812 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.825978 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.827434 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.828217 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.829034 (+ 817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.829165 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.829209 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.829304 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":90,"threads_started":1}
I20260810 23:06:15.831997 15359 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac95ca52b884e77a18cbe4b06dc00ae (DEFAULT_TABLE table=test-table [id=2fa9830a8e4943d9bd91f2ab2f541ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:15.831831 15642 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac95ca52b884e77a18cbe4b06dc00ae (DEFAULT_TABLE table=test-table [id=2fa9830a8e4943d9bd91f2ab2f541ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:15.832947 15359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac95ca52b884e77a18cbe4b06dc00ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:15.832991 15642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac95ca52b884e77a18cbe4b06dc00ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:15.832828 15497 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac95ca52b884e77a18cbe4b06dc00ae (DEFAULT_TABLE table=test-table [id=2fa9830a8e4943d9bd91f2ab2f541ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:15.833853 15497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac95ca52b884e77a18cbe4b06dc00ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:15.840342 15732 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: Bootstrap starting.
I20260810 23:06:15.841048 15733 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Bootstrap starting.
I20260810 23:06:15.841734 15734 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Bootstrap starting.
I20260810 23:06:15.843225 15733 tablet_bootstrap.cc:654] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:15.843247 15732 tablet_bootstrap.cc:654] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:15.843513 15734 tablet_bootstrap.cc:654] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:15.844560 15734 log.cc:824] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:15.844627 15732 log.cc:824] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:15.844627 15733 log.cc:824] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:15.846794 15734 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: No bootstrap required, opened a new log
I20260810 23:06:15.846884 15732 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: No bootstrap required, opened a new log
I20260810 23:06:15.846896 15733 tablet_bootstrap.cc:492] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: No bootstrap required, opened a new log
I20260810 23:06:15.847025 15734 ts_tablet_manager.cc:1397] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:06:15.847091 15732 ts_tablet_manager.cc:1397] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:06:15.847123 15733 ts_tablet_manager.cc:1397] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:06:15.850131 15734 raft_consensus.cc:348] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.850412 15734 raft_consensus.cc:374] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:15.850477 15734 raft_consensus.cc:729] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5478222305b429b99f898090d83fe5e, State: Initialized, Role: FOLLOWER
I20260810 23:06:15.851043 15734 consensus_queue.cc:260] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.851054 15732 raft_consensus.cc:348] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.851054 15733 raft_consensus.cc:348] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.851322 15732 raft_consensus.cc:374] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:15.851322 15733 raft_consensus.cc:374] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:15.851387 15732 raft_consensus.cc:729] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0e500d623c450bb0bd7644c5eef5c8, State: Initialized, Role: FOLLOWER
I20260810 23:06:15.851387 15733 raft_consensus.cc:729] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b291f61f2ee9431d8086055980a9061d, State: Initialized, Role: FOLLOWER
I20260810 23:06:15.851826 15732 consensus_queue.cc:260] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.851855 15733 consensus_queue.cc:260] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.851934 15571 heartbeater.cc:503] Master 127.14.75.62:46729 was elected leader, sending a full tablet report...
I20260810 23:06:15.852157 15734 ts_tablet_manager.cc:1428] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:06:15.852761 15732 ts_tablet_manager.cc:1428] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:06:15.852770 15733 ts_tablet_manager.cc:1428] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:06:15.853081 15714 heartbeater.cc:503] Master 127.14.75.62:46729 was elected leader, sending a full tablet report...
I20260810 23:06:15.853081 15429 heartbeater.cc:503] Master 127.14.75.62:46729 was elected leader, sending a full tablet report...
I20260810 23:06:15.989840 15739 raft_consensus.cc:482] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:15.990072 15739 raft_consensus.cc:504] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.991276 15739 leader_election.cc:288] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5478222305b429b99f898090d83fe5e (127.14.75.2:42689), 7d0e500d623c450bb0bd7644c5eef5c8 (127.14.75.1:39677)
I20260810 23:06:15.995452 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.991658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:34189 (local address 127.14.75.2:42689)
0810 23:06:15.991792 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.991797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.991814 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:15.991944 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.991951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.992019 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.992114 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:15.992424 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.993160 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.994396 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.994819 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.994942 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.995012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.995219 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.995308 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:15.995452 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.991555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:42689 (local address 127.14.75.3:34189)
0810 23:06:15.991718 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.991741 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.991829 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.992140 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.992143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.992163 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:15.992296 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.992306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.993267 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.993269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.994260 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.994271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.994968 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.994972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.995095 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.995114 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.995201 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.995329 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":37}
I20260810 23:06:15.995630 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.991570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39677 (local address 127.14.75.3:40101)
0810 23:06:15.992277 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:15.992290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:15.992392 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:15.992645 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:15.992647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:15.992661 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:15.992755 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.992760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.993673 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.993679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.994479 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:15.994489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.995149 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.995152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:15.995288 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.995306 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:06:15.995390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:06:15.995461 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":268,"thread_start_us":154,"threads_started":1}
I20260810 23:06:15.995613 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:15.991690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40101 (local address 127.14.75.1:39677)
0810 23:06:15.991830 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:15.991837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:15.992400 (+ 563us) server_negotiation.cc:407] Connection header received
0810 23:06:15.992510 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:15.992512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:15.992559 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:15.992631 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:15.992854 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.993563 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.994615 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:15.995029 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:15.995144 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:15.995209 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:15.995408 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 23:06:15.995487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:06:15.996326 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac95ca52b884e77a18cbe4b06dc00ae" candidate_uuid: "b291f61f2ee9431d8086055980a9061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" is_pre_election: true
I20260810 23:06:15.996275 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac95ca52b884e77a18cbe4b06dc00ae" candidate_uuid: "b291f61f2ee9431d8086055980a9061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5478222305b429b99f898090d83fe5e" is_pre_election: true
I20260810 23:06:15.996654 15521 raft_consensus.cc:2454] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b291f61f2ee9431d8086055980a9061d in term 0.
I20260810 23:06:15.996658 15381 raft_consensus.cc:2454] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b291f61f2ee9431d8086055980a9061d in term 0.
I20260810 23:06:15.997208 15595 leader_election.cc:302] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7d0e500d623c450bb0bd7644c5eef5c8, b291f61f2ee9431d8086055980a9061d, c5478222305b429b99f898090d83fe5e; no voters:
I20260810 23:06:15.997578 15739 raft_consensus.cc:2781] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:15.997663 15739 raft_consensus.cc:482] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:15.997709 15739 raft_consensus.cc:3037] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:15.998824 15739 raft_consensus.cc:504] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:15.999218 15739 leader_election.cc:288] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [CANDIDATE]: Term 1 election: Requested vote from peers c5478222305b429b99f898090d83fe5e (127.14.75.2:42689), 7d0e500d623c450bb0bd7644c5eef5c8 (127.14.75.1:39677)
I20260810 23:06:15.999632 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac95ca52b884e77a18cbe4b06dc00ae" candidate_uuid: "b291f61f2ee9431d8086055980a9061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0e500d623c450bb0bd7644c5eef5c8"
I20260810 23:06:15.999682 15521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac95ca52b884e77a18cbe4b06dc00ae" candidate_uuid: "b291f61f2ee9431d8086055980a9061d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5478222305b429b99f898090d83fe5e"
I20260810 23:06:15.999783 15381 raft_consensus.cc:3037] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:15.999823 15521 raft_consensus.cc:3037] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:16.000928 15381 raft_consensus.cc:2454] T 9ac95ca52b884e77a18cbe4b06dc00ae P 7d0e500d623c450bb0bd7644c5eef5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b291f61f2ee9431d8086055980a9061d in term 1.
I20260810 23:06:16.001269 15521 raft_consensus.cc:2454] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b291f61f2ee9431d8086055980a9061d in term 1.
I20260810 23:06:16.001315 15592 leader_election.cc:302] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0e500d623c450bb0bd7644c5eef5c8, b291f61f2ee9431d8086055980a9061d; no voters:
I20260810 23:06:16.001526 15739 raft_consensus.cc:2781] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:16.001811 15739 raft_consensus.cc:686] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [term 1 LEADER]: Becoming Leader. State: Replica: b291f61f2ee9431d8086055980a9061d, State: Running, Role: LEADER
I20260810 23:06:16.002168 15739 consensus_queue.cc:237] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } }
I20260810 23:06:16.005353 15229 catalog_manager.cc:5495] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d reported cstate change: term changed from 0 to 1, leader changed from <none> to b291f61f2ee9431d8086055980a9061d (127.14.75.3). New cstate: current_term: 1 leader_uuid: "b291f61f2ee9431d8086055980a9061d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b291f61f2ee9431d8086055980a9061d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5478222305b429b99f898090d83fe5e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 42689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d0e500d623c450bb0bd7644c5eef5c8" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39677 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:16.024005 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:16.014348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.14.75.1:39677)
0810 23:06:16.014521 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:16.014526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:16.014658 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:06:16.014819 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:16.014822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:16.014881 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:16.014961 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:16.015697 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.016213 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.017186 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.017480 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.017596 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:16.020242 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:16.020257 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:16.020283 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:16.020310 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:16.022169 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.022650 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.022656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.022658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.022748 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.023029 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.023033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.023035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.023307 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:16.023545 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:16.023700 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:06:16.023837 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:06:16.026706 15716 tablet.cc:2369] T 9ac95ca52b884e77a18cbe4b06dc00ae P b291f61f2ee9431d8086055980a9061d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:16.053206 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:16.043873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.14.75.1:39677)
0810 23:06:16.044136 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:16.044141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:16.044258 (+ 117us) server_negotiation.cc:407] Connection header received
0810 23:06:16.044448 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:16.044452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:16.044502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:16.044594 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:16.045299 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.045920 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.046767 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.047000 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.047122 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:16.049841 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:16.049844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:16.049846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:16.049881 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:16.051424 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.051865 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.051870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.051872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.051964 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.052254 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.052257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.052259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.052450 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:16.052585 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:16.052886 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:06:16.053035 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260810 23:06:16.058818 15572 tablet.cc:2369] T 9ac95ca52b884e77a18cbe4b06dc00ae P c5478222305b429b99f898090d83fe5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:16.080730 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:16.072058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.14.75.1:39677)
0810 23:06:16.072228 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:16.072233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:16.072360 (+ 127us) server_negotiation.cc:407] Connection header received
0810 23:06:16.072536 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:16.072545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:16.072596 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:16.072682 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:16.073433 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.073948 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.074854 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:16.075082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:16.075187 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:16.077767 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:16.077770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:16.077772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:16.077798 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:16.079264 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.079723 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.079726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.079727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.079799 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:16.080016 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:16.080018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:16.080019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:16.080185 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:16.080281 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:16.080492 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 23:06:16.080607 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 23:06:16.082158 15338 tablet_service.cc:2908] Rejecting scan request for tablet 9ac95ca52b884e77a18cbe4b06dc00ae: Uninitialized: clean time has not yet been initialized
I20260810 23:06:16.084311 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15294
I20260810 23:06:16.091185 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15434
I20260810 23:06:16.097877 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15576
I20260810 23:06:16.104939 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15198
2026-08-10T23:06:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2441 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15756](info): set up 2 sockets
Aug 10 23:06:16 dist-test-slave-srgv krb5kdc[15756](info): commencing operation
krb5kdc: starting...
W20260810 23:06:17.157980 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:17.285826 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41569
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:17.401265 15769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:17.401681 15769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:17.401734 15769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:17.405804 15769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:17.405891 15769 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:06:17.405910 15769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:17.405927 15769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:17.405944 15769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:17.405961 15769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:17.411429 15769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41569
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:17.412652 15769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:17.413697 15769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:17.422401 15775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.422401 15774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.422438 15777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:17.422459 15769 server_base.cc:1034] running on GCE node
I20260810 23:06:17.423331 15769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:17.424551 15769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:17.425755 15769 hybrid_clock.cc:648] HybridClock initialized: now 1786403177425737 us; error 34 us; skew 500 ppm
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:17.428886 15769 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:17.429893 15769 webserver.cc:460] Webserver started at http://127.14.75.62:36115/ using document root <none> and password file <none>
I20260810 23:06:17.430441 15769 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:17.430505 15769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:17.430681 15769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:17.432785 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "00a95867213647d380cb52d06aac8810"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "52e679cbafb8912e1b6bd061210842c0"
server_key_iv: "ed9a3cc7cba2bdc1fa7ae3c9bc752c94"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.433326 15769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "00a95867213647d380cb52d06aac8810"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "52e679cbafb8912e1b6bd061210842c0"
server_key_iv: "ed9a3cc7cba2bdc1fa7ae3c9bc752c94"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.437251 15769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:06:17.439774 15785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:17.441022 15769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:17.441138 15769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "00a95867213647d380cb52d06aac8810"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "52e679cbafb8912e1b6bd061210842c0"
server_key_iv: "ed9a3cc7cba2bdc1fa7ae3c9bc752c94"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.441236 15769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:17.462858 15769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:17.466547 15769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:17.466727 15769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:17.474996 15769 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41569
I20260810 23:06:17.475067 15851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41569 every 8 connection(s)
I20260810 23:06:17.476011 15769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:17.478957 15852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:17.483081 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15769
I20260810 23:06:17.483243 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:17.483506 14636 external_mini_cluster.cc:1428] Setting key 78cc53e18592bb043141fa4b0b2268ea
I20260810 23:06:17.485159 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Bootstrap starting.
I20260810 23:06:17.487506 15852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:17.488283 15852 log.cc:824] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:17.490595 15852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: No bootstrap required, opened a new log
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:17.494354 15852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:06:17.494616 15852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:17.494691 15852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a95867213647d380cb52d06aac8810, State: Initialized, Role: FOLLOWER
I20260810 23:06:17.495157 15852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:06:17.495314 15852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:17.495386 15852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:17.495507 15852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:17.496449 15852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:06:17.496835 15852 leader_election.cc:302] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a95867213647d380cb52d06aac8810; no voters:
I20260810 23:06:17.497215 15852 leader_election.cc:288] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:17.497531 15857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:17.497759 15857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [term 1 LEADER]: Becoming Leader. State: Replica: 00a95867213647d380cb52d06aac8810, State: Running, Role: LEADER
I20260810 23:06:17.498098 15857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:06:17.498487 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.485032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.14.75.62:41569)
0810 23:06:17.485562 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:17.485579 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:17.485619 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:06:17.486350 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:17.486369 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:17.486725 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:17.487074 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:17.487939 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.488987 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.489922 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.490395 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.490568 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.493086 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:17.493091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:17.493105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:17.493134 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:17.495206 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.495678 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.495688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.495693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.495816 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.496101 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.496105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.496107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.496524 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:17.496734 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:17.497050 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:06:17.497390 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":166,"threads_started":1}
I20260810 23:06:17.498534 15852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:17.499354 15857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a95867213647d380cb52d06aac8810. Latest consensus state: current_term: 1 leader_uuid: "00a95867213647d380cb52d06aac8810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } } }
I20260810 23:06:17.499477 15857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:17.499781 15858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a95867213647d380cb52d06aac8810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a95867213647d380cb52d06aac8810" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } } }
I20260810 23:06:17.499878 15858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810 [sys.catalog]: This master's current role is: LEADER
W20260810 23:06:17.503656 15872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:17.503758 15872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:17.503856 15866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:17.504619 15866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:17.510311 15866 catalog_manager.cc:1347] Generated new cluster ID: a32eb289f6b54214a169456ae0d4d9d3
I20260810 23:06:17.510398 15866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:17.522459 15866 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:17.523586 15866 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:17.528014 15866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Generated new TSK 0
I20260810 23:06:17.528787 15866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:17.558559 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:17.677858 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:17.678179 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:17.678227 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:17.682528 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:17.682612 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:17.682654 15878 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:17.682693 15878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:17.688498 15878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:17.689774 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:17.690836 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:17.701202 15878 server_base.cc:1034] running on GCE node
W20260810 23:06:17.701086 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.701100 15886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.701056 15883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:17.701845 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:17.702499 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:17.703692 15878 hybrid_clock.cc:648] HybridClock initialized: now 1786403177703671 us; error 34 us; skew 500 ppm
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:17.711663 15878 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:17.712754 15878 webserver.cc:460] Webserver started at http://127.14.75.1:40561/ using document root <none> and password file <none>
I20260810 23:06:17.713457 15878 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:17.713567 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:17.713824 15878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:17.716080 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "40bab2a6360743f9bddcee440fc15446"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "c501453d1d04532adc92d75c288d7ef1"
server_key_iv: "657edaf5306e451a56f513e821859c69"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.716715 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "40bab2a6360743f9bddcee440fc15446"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "c501453d1d04532adc92d75c288d7ef1"
server_key_iv: "657edaf5306e451a56f513e821859c69"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.720860 15878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:17.723546 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:17.724848 15878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:06:17.725000 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "40bab2a6360743f9bddcee440fc15446"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "c501453d1d04532adc92d75c288d7ef1"
server_key_iv: "657edaf5306e451a56f513e821859c69"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.725127 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:17.766834 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:17.770500 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:17.770730 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:17.772181 15878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:17.772229 15878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:17.772307 15878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:17.772344 15878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:17.783674 15878 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:38573
I20260810 23:06:17.783993 16013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:38573 every 8 connection(s)
I20260810 23:06:17.784670 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:17.786473 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 15878
I20260810 23:06:17.786605 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:17.786881 14636 external_mini_cluster.cc:1428] Setting key ef2b6f17372e7900f6b8fd7602a754db
I20260810 23:06:17.787505 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.773121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:41175 (local address 127.14.75.62:41569)
0810 23:06:17.773253 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:17.773257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:17.774375 (+ 1118us) server_negotiation.cc:407] Connection header received
0810 23:06:17.775578 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:17.775583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:17.775653 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:17.775766 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:17.777733 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.778646 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.779843 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.780286 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.780421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.783267 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:17.783271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:17.783274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:17.783307 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:17.785061 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.785729 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.785733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.785737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.785838 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.786199 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.786221 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.786223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.786497 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:17.786607 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:17.787199 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 23:06:17.787355 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:06:17.788455 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.773727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.1:41175)
0810 23:06:17.774222 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:17.774267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:17.775322 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:17.775979 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:17.775990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:17.776512 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:17.777533 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.777556 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.778798 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.778802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.779728 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.779738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.780453 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.780457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.780623 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.781284 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:17.781301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:17.783054 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:17.785194 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:17.785200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:17.785215 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:17.785596 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:17.785949 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:17.785952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:17.785954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:17.786096 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:17.786636 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:17.786643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:17.786911 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 23:06:17.787125 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 23:06:17.787474 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260810 23:06:17.790714 16014 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
I20260810 23:06:17.790904 16014 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:17.791543 16014 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:06:17.793264 15801 ts_manager.cc:194] Registered new tserver with Master: 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573)
I20260810 23:06:17.794459 15801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:41175
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:17.803977 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.792483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.1:51281)
0810 23:06:17.792805 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:17.792823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:17.792914 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:17.795573 (+ 2659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:17.795576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:17.795849 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:17.796029 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.796041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.797082 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.797085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.798015 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.798026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.798603 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.798609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.798717 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.799316 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:17.799329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:17.799816 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:17.802057 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:17.802063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:17.802069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:17.802365 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:17.802745 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:17.802747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:17.802748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:17.802833 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:17.803293 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:17.803295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:17.803583 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 23:06:17.803691 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:06:17.803827 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":113,"threads_started":1}
I20260810 23:06:17.804059 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.795058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:51281 (local address 127.14.75.62:41569)
0810 23:06:17.795212 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:17.795218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:17.795242 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:06:17.795289 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:17.795293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:17.795350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:17.795440 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:17.796185 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.796932 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.798146 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.798472 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.798556 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.800240 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:17.800244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:17.800247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:17.800276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:17.801940 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.802502 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.802508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.802511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.802623 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:17.802926 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:17.802931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:17.802934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:17.803195 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:17.803307 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:17.803792 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 23:06:17.803908 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:17.816821 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:17.931325 16024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:17.931619 16024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:17.931697 16024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:17.935714 16024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:17.935802 16024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:17.935874 16024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:17.935950 16024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:17.941311 16024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:17.942412 16024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:17.943440 16024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:17.954209 16029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.954193 16030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:17.954355 16032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:17.955178 16024 server_base.cc:1034] running on GCE node
I20260810 23:06:17.955613 16024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:17.956267 16024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:17.957460 16024 hybrid_clock.cc:648] HybridClock initialized: now 1786403177957444 us; error 29 us; skew 500 ppm
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:17.960198 16024 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:17.961105 16024 webserver.cc:460] Webserver started at http://127.14.75.2:41889/ using document root <none> and password file <none>
I20260810 23:06:17.961711 16024 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:17.961786 16024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:17.962026 16024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:17.964093 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "d80f0d6b8cd3b83328ac5571c34a0c19"
server_key_iv: "48b65267fb1fac64b544af80031d1627"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.964649 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "d80f0d6b8cd3b83328ac5571c34a0c19"
server_key_iv: "48b65267fb1fac64b544af80031d1627"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.968518 16024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:17.971024 16039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:17.972194 16024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 23:06:17.972330 16024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c"
format_stamp: "Formatted at 2026-08-10 23:06:17 on dist-test-slave-srgv"
server_key: "d80f0d6b8cd3b83328ac5571c34a0c19"
server_key_iv: "48b65267fb1fac64b544af80031d1627"
server_key_version: "encryptionkey@0"
I20260810 23:06:17.972445 16024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:17.986245 16024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:17.989243 16024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:17.989511 16024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:17.991081 16024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:17.991158 16024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:17.991226 16024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:17.991269 16024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:17 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:18.002911 16024 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:35417
I20260810 23:06:18.003294 16159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:35417 every 8 connection(s)
I20260810 23:06:18.003927 16024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:18.006129 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.991944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:37021 (local address 127.14.75.62:41569)
0810 23:06:17.992124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:17.992131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:17.992988 (+ 857us) server_negotiation.cc:407] Connection header received
0810 23:06:17.994289 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:17.994296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:17.994383 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:17.994487 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:17.995902 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.996436 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.997900 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.998120 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.998244 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.002190 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:18.002197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:18.002203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:18.002249 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:18.003993 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.004631 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.004634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.004636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.004740 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.005098 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.005101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.005102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.005292 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:18.005368 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.005903 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.006011 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:06:18.006897 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:17.992310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.2:37021)
0810 23:06:17.992797 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:17.992859 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:17.994038 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:17.994663 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:17.994670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:17.995193 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:17.995749 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.995766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.996564 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.996567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.997774 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:17.997784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:17.998302 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:17.998306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:17.998478 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:17.999216 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:17.999236 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:18.001998 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:18.004130 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.004144 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.004161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.004503 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.004841 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.004845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.004846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.005005 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.005379 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:18.005384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:18.005674 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.005834 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.006092 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260810 23:06:18.008001 16160 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
I20260810 23:06:18.008147 16160 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:18.008723 16160 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:06:18.009850 15802 ts_manager.cc:194] Registered new tserver with Master: 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417)
I20260810 23:06:18.010514 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:37021
I20260810 23:06:18.014097 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16024
I20260810 23:06:18.014215 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:18.014535 14636 external_mini_cluster.cc:1428] Setting key f2252741a6f9921902867f5be9602633
I20260810 23:06:18.017799 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.010445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:40839 (local address 127.14.75.62:41569)
0810 23:06:18.010572 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.010579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.010729 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:06:18.010857 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.010860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.010904 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.010996 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:18.011478 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.011957 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.012912 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.013108 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.013180 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.014448 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:18.014451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:18.014453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:18.014472 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:18.016101 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.016579 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.016582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.016583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.016657 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.016957 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.016960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.016961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.017163 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:18.017239 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.017570 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.017678 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:18.018080 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.010337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.2:40839)
0810 23:06:18.010650 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.010663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.010766 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.011011 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.011013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.011255 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:18.011369 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.011377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.012086 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.012089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.012813 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.012823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.013195 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.013197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.013275 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.013846 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:18.013866 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:18.014327 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:18.016214 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.016217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.016219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.016484 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.016742 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.016747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.016752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.016863 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.017256 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:18.017262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:18.017379 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.017525 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.017707 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:18.045897 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:18.189634 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:18.189970 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:18.190054 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:18.194087 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:18.194190 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:18.194249 16170 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:18.194324 16170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:18.199815 16170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:18.200960 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:18.202030 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:18.211627 16178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:18.211645 16176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:18.211654 16175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:18.211985 16170 server_base.cc:1034] running on GCE node
I20260810 23:06:18.212417 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:18.213090 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:18.214270 16170 hybrid_clock.cc:648] HybridClock initialized: now 1786403178214256 us; error 27 us; skew 500 ppm
Aug 10 23:06:18 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403178, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:18.217104 16170 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:18.217969 16170 webserver.cc:460] Webserver started at http://127.14.75.3:45921/ using document root <none> and password file <none>
I20260810 23:06:18.218515 16170 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:18.218576 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:18.218745 16170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:18.220849 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "43c3723e1076480b88efea4c605a9e31"
format_stamp: "Formatted at 2026-08-10 23:06:18 on dist-test-slave-srgv"
server_key: "d991a67141d022ac6ad4e04df13cf808"
server_key_iv: "401080b82211112b1ed53b08a933fbe4"
server_key_version: "encryptionkey@0"
I20260810 23:06:18.221372 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "43c3723e1076480b88efea4c605a9e31"
format_stamp: "Formatted at 2026-08-10 23:06:18 on dist-test-slave-srgv"
server_key: "d991a67141d022ac6ad4e04df13cf808"
server_key_iv: "401080b82211112b1ed53b08a933fbe4"
server_key_version: "encryptionkey@0"
I20260810 23:06:18.225018 16170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:18.227509 16185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:18.228734 16170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:18.228858 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "43c3723e1076480b88efea4c605a9e31"
format_stamp: "Formatted at 2026-08-10 23:06:18 on dist-test-slave-srgv"
server_key: "d991a67141d022ac6ad4e04df13cf808"
server_key_iv: "401080b82211112b1ed53b08a933fbe4"
server_key_version: "encryptionkey@0"
I20260810 23:06:18.228971 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:18.256556 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:18.260125 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:18.260305 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:18.261847 16170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:18.261936 16170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:18.262009 16170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:18.262050 16170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:18 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403178, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:18.273689 16170 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:36841
I20260810 23:06:18.273731 16307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:36841 every 8 connection(s)
I20260810 23:06:18.274859 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:18.276342 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.262606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:47127 (local address 127.14.75.62:41569)
0810 23:06:18.262732 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.262736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.264426 (+ 1690us) server_negotiation.cc:407] Connection header received
0810 23:06:18.265643 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.265646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.265706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.265809 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:18.267217 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.267758 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.269007 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.269234 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.269347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.272298 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:18.272301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:18.272304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:18.272335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:18.274103 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.274598 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.274602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.274603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.274701 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.275038 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.275041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.275043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.275337 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:18.275440 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.276103 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.276208 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:06:18.277207 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.263710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.3:47127)
0810 23:06:18.264278 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.264323 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.265361 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.265987 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.265993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.266584 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:18.267065 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.267078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.267881 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.267885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.268888 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.268897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.269399 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.269403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.269581 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.270331 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:18.270347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:18.272125 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:18.274216 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.274221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.274234 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.274484 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.274805 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.274807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.274808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.274936 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.275498 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:18.275506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:18.275822 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.276053 (+ 231us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.276328 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":145,"threads_started":1}
I20260810 23:06:18.278553 16308 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
I20260810 23:06:18.278719 16308 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:18.279222 16308 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:06:18.280222 15802 ts_manager.cc:194] Registered new tserver with Master: 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841)
I20260810 23:06:18.280965 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:47127
I20260810 23:06:18.284287 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16170
I20260810 23:06:18.284415 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:18.284684 14636 external_mini_cluster.cc:1428] Setting key f3bb8c5b6bfa088640feca67db16d222
I20260810 23:06:18.286939 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:18.288017 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.280633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:33571 (local address 127.14.75.62:41569)
0810 23:06:18.280752 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.280757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.280925 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:06:18.281040 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.281043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.281108 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.281193 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:18.281677 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.282171 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.282986 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.283194 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.283292 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.284527 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:18.284530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:18.284532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:18.284554 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:18.286169 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.286705 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.286709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.286712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.286820 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.287153 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.287158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.287160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.287363 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:18.287469 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.287739 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.287852 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:06:18.288033 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.280563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.3:33571)
0810 23:06:18.280850 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.280862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.280965 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.281208 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.281211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.281464 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:18.281573 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.281579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.282295 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.282298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.282850 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.282858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.283305 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.283309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.283397 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.283875 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:18.283886 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:18.284334 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:18.286288 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.286292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.286294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.286578 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.286923 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:18.286927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:18.286929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:18.287055 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:18.287474 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:18.287478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:18.287575 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.287681 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.287856 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
Aug 10 23:06:18 dist-test-slave-srgv krb5kdc[15756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:06:18 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:18.313910 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.305568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.14.75.62:41569)
0810 23:06:18.305700 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.305704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.305715 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:18.305763 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.305765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.305804 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.305872 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:18.306595 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.307115 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.308023 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.308234 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.308315 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.310916 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:18.310920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:18.310922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:18.310945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:18.312181 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.312602 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.312605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.312607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.312678 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:18.312951 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:18.312955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:18.312957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:18.313179 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:18.313329 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.313611 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.313746 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:06:18.316998 15802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:18.320137 15802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:18.334964 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.328446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.14.75.1:38573)
0810 23:06:18.329924 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.329934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.329957 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:06:18.330033 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.330038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.330219 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.330377 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:18.330743 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.331861 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.333306 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334192 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.334308 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.334461 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.334682 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.334798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":78,"threads_started":1}
I20260810 23:06:18.334980 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.328321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:38573 (local address 127.0.0.1:33418)
0810 23:06:18.328833 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.328882 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.329026 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.330368 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.330375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.330406 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:18.330594 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.330603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.331994 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.331997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.333186 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.333196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334329 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.334333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.334460 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.334505 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.334626 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.334793 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":83,"spinlock_wait_cycles":7808,"thread_start_us":104,"threads_started":1}
I20260810 23:06:18.335175 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.328460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35417 (local address 127.0.0.1:34768)
0810 23:06:18.328833 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.328882 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.329025 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.331099 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.331102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.331121 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:18.331229 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.331235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.332566 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.332569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.333630 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.333641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334711 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.334714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.334870 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.334892 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.334956 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.335040 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":33,"spinlock_wait_cycles":12544,"thread_start_us":123,"threads_started":1}
I20260810 23:06:18.335841 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.329535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.14.75.2:35417)
0810 23:06:18.330677 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.330691 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.330714 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:06:18.330785 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.330789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.330955 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.331100 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:18.331350 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.332440 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.333766 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334557 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.335440 (+ 883us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.335560 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.335597 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.335690 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":133,"threads_started":1}
I20260810 23:06:18.335969 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.329231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.14.75.3:36841)
0810 23:06:18.329867 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.329881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.329907 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:06:18.329996 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.330000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.330222 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.330377 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:18.330711 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.331669 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.333787 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334358 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.334461 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.334625 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.335718 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.335818 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":132,"threads_started":1}
I20260810 23:06:18.336215 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.329116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36841 (local address 127.0.0.1:52770)
0810 23:06:18.329526 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.329538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.329620 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.330369 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.330375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.330413 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:18.330591 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.330598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.331774 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.331779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.333602 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.333610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.334684 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.334687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.334808 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.334862 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.334930 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.334994 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":527,"spinlock_wait_cycles":10368,"thread_start_us":125,"threads_started":1}
I20260810 23:06:18.338342 16091 tablet_service.cc:1467] Processing CreateTablet for tablet e13d65a898df4a339bef25d127f46867 (DEFAULT_TABLE table=test-table [id=9f755f3ba9374bcab2b49bcc7b066293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.338340 15944 tablet_service.cc:1467] Processing CreateTablet for tablet e13d65a898df4a339bef25d127f46867 (DEFAULT_TABLE table=test-table [id=9f755f3ba9374bcab2b49bcc7b066293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.338543 16237 tablet_service.cc:1467] Processing CreateTablet for tablet e13d65a898df4a339bef25d127f46867 (DEFAULT_TABLE table=test-table [id=9f755f3ba9374bcab2b49bcc7b066293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.339459 16237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13d65a898df4a339bef25d127f46867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.339466 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13d65a898df4a339bef25d127f46867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.339466 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13d65a898df4a339bef25d127f46867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.346621 16333 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Bootstrap starting.
I20260810 23:06:18.347518 16334 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Bootstrap starting.
I20260810 23:06:18.349066 16333 tablet_bootstrap.cc:654] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.349216 16334 tablet_bootstrap.cc:654] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.349753 16335 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Bootstrap starting.
I20260810 23:06:18.349970 16333 log.cc:824] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:18.350301 16334 log.cc:824] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:18.351867 16335 tablet_bootstrap.cc:654] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.352173 16333 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: No bootstrap required, opened a new log
I20260810 23:06:18.352277 16334 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: No bootstrap required, opened a new log
I20260810 23:06:18.352384 16333 ts_tablet_manager.cc:1397] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:06:18.352461 16334 ts_tablet_manager.cc:1397] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 23:06:18.352674 16335 log.cc:824] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:18.354709 16335 tablet_bootstrap.cc:492] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: No bootstrap required, opened a new log
I20260810 23:06:18.354897 16335 ts_tablet_manager.cc:1397] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:06:18.355638 16333 raft_consensus.cc:348] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.355883 16333 raft_consensus.cc:374] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.355937 16333 raft_consensus.cc:729] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c3723e1076480b88efea4c605a9e31, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.356365 16333 consensus_queue.cc:260] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.357035 16308 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:06:18.357328 16333 ts_tablet_manager.cc:1428] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:06:18.358235 16334 raft_consensus.cc:348] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.358517 16334 raft_consensus.cc:374] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.358577 16334 raft_consensus.cc:729] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bab2a6360743f9bddcee440fc15446, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.359038 16334 consensus_queue.cc:260] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.359037 16335 raft_consensus.cc:348] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.359278 16335 raft_consensus.cc:374] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.359340 16335 raft_consensus.cc:729] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bf63ed5f3d48f59e2889ffa2b26a5c, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.359743 16335 consensus_queue.cc:260] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.360069 16334 ts_tablet_manager.cc:1428] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260810 23:06:18.360428 16014 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:06:18.360586 16160 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:06:18.360688 16335 ts_tablet_manager.cc:1428] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 23:06:18.505123 16161 tablet.cc:2369] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:18.526032 16309 tablet.cc:2369] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:18.536494 16015 tablet.cc:2369] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:18.713349 16342 raft_consensus.cc:482] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:18.713620 16342 raft_consensus.cc:504] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.714731 16342 leader_election.cc:288] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573), 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841)
I20260810 23:06:18.718923 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.715000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36841 (local address 127.14.75.2:33561)
0810 23:06:18.715481 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.715507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.715674 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.715986 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.715989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.716009 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:18.716114 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.716119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.716875 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.716877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.717795 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.717805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.718454 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.718457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.718555 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.718598 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.718684 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.718765 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":52,"thread_start_us":76,"threads_started":1}
I20260810 23:06:18.718937 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.715131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:33561 (local address 127.14.75.3:36841)
0810 23:06:18.715297 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.715302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.715625 (+ 323us) server_negotiation.cc:407] Connection header received
0810 23:06:18.715814 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.715816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.715861 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.715954 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:18.716246 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.716741 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.717940 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.718327 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.718436 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.718502 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.718727 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.718811 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:06:18.718937 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.715214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:33987 (local address 127.14.75.1:38573)
0810 23:06:18.715398 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.715403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.715625 (+ 222us) server_negotiation.cc:407] Connection header received
0810 23:06:18.715783 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.715788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.715842 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.715904 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:18.716198 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.716716 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.717905 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.718339 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.718460 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.718523 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.718765 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.718834 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:06:18.719316 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.715000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:38573 (local address 127.14.75.2:33987)
0810 23:06:18.715487 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:18.715505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:18.715659 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:18.715945 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:18.715948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:18.715964 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:18.716085 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.716097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.716833 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.716836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.717778 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:18.717788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.718477 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.718481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:18.718604 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.718622 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:06:18.719077 (+ 455us) client_negotiation.cc:240] Negotiation successful
0810 23:06:18.719179 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":83,"thread_start_us":109,"threads_started":1}
I20260810 23:06:18.719626 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13d65a898df4a339bef25d127f46867" candidate_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c3723e1076480b88efea4c605a9e31" is_pre_election: true
I20260810 23:06:18.719929 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13d65a898df4a339bef25d127f46867" candidate_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bab2a6360743f9bddcee440fc15446" is_pre_election: true
I20260810 23:06:18.720000 16259 raft_consensus.cc:2454] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74bf63ed5f3d48f59e2889ffa2b26a5c in term 0.
I20260810 23:06:18.720212 15966 raft_consensus.cc:2454] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74bf63ed5f3d48f59e2889ffa2b26a5c in term 0.
I20260810 23:06:18.720568 16043 leader_election.cc:302] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c3723e1076480b88efea4c605a9e31, 74bf63ed5f3d48f59e2889ffa2b26a5c; no voters:
I20260810 23:06:18.720878 16342 raft_consensus.cc:2781] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:18.720978 16342 raft_consensus.cc:482] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:18.721040 16342 raft_consensus.cc:3037] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:18.722182 16342 raft_consensus.cc:504] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.722543 16342 leader_election.cc:288] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [CANDIDATE]: Term 1 election: Requested vote from peers 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573), 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841)
I20260810 23:06:18.722950 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13d65a898df4a339bef25d127f46867" candidate_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bab2a6360743f9bddcee440fc15446"
I20260810 23:06:18.722950 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13d65a898df4a339bef25d127f46867" candidate_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c3723e1076480b88efea4c605a9e31"
I20260810 23:06:18.723088 15966 raft_consensus.cc:3037] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:18.723099 16259 raft_consensus.cc:3037] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:18.724539 15966 raft_consensus.cc:2454] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bf63ed5f3d48f59e2889ffa2b26a5c in term 1.
I20260810 23:06:18.724877 16040 leader_election.cc:302] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bab2a6360743f9bddcee440fc15446, 74bf63ed5f3d48f59e2889ffa2b26a5c; no voters:
I20260810 23:06:18.724974 16259 raft_consensus.cc:2454] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bf63ed5f3d48f59e2889ffa2b26a5c in term 1.
I20260810 23:06:18.725098 16342 raft_consensus.cc:2781] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:18.725383 16342 raft_consensus.cc:686] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 LEADER]: Becoming Leader. State: Replica: 74bf63ed5f3d48f59e2889ffa2b26a5c, State: Running, Role: LEADER
I20260810 23:06:18.725749 16342 consensus_queue.cc:237] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.729028 15802 catalog_manager.cc:5495] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2). New cstate: current_term: 1 leader_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:18.793767 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:18.789949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.14.75.2:35417)
0810 23:06:18.790087 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:18.790091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:18.790107 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:18.790320 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:18.790323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:18.790382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:18.790466 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:18.790773 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.791277 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.792180 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:18.792401 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:18.792507 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:18.792870 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:18.793281 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:18.793400 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:18.793551 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:06:18.793616 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:06:18.799707 15966 raft_consensus.cc:1261] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Refusing update from remote peer 74bf63ed5f3d48f59e2889ffa2b26a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:18.799707 16259 raft_consensus.cc:1261] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Refusing update from remote peer 74bf63ed5f3d48f59e2889ffa2b26a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:18.800480 16345 consensus_queue.cc:1035] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:18.800711 16342 consensus_queue.cc:1035] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:18.808281 16349 mvcc.cc:205] Tried to move back new op lower bound from 7317107420356030464 to 7317107420063260672. Current Snapshot: MvccSnapshot[applied={T|T < 7317107420356030464}]
I20260810 23:06:18.809100 16350 mvcc.cc:205] Tried to move back new op lower bound from 7317107420356030464 to 7317107420063260672. Current Snapshot: MvccSnapshot[applied={T|T < 7317107420356030464}]
I20260810 23:06:18.823307 15800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:18.823634 15800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:18.826680 15800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Sending DeleteTablet for 3 replicas of tablet e13d65a898df4a339bef25d127f46867
I20260810 23:06:18.827725 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet e13d65a898df4a339bef25d127f46867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:18 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:33418
I20260810 23:06:18.827745 16091 tablet_service.cc:1514] Processing DeleteTablet for tablet e13d65a898df4a339bef25d127f46867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:18 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:34768
I20260810 23:06:18.828507 16360 tablet_replica.cc:330] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: stopping tablet replica
I20260810 23:06:18.828487 16237 tablet_service.cc:1514] Processing DeleteTablet for tablet e13d65a898df4a339bef25d127f46867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:18 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260810 23:06:18.828425 15800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:06:18.828886 16361 tablet_replica.cc:330] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: stopping tablet replica
W20260810 23:06:18.828882 15800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:18.829138 16361 raft_consensus.cc:2229] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:18.829303 16360 raft_consensus.cc:2229] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:18.829399 16361 raft_consensus.cc:2258] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:18.829897 16360 raft_consensus.cc:2258] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:18.830366 16359 tablet_replica.cc:330] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: stopping tablet replica
I20260810 23:06:18.830667 16359 raft_consensus.cc:2229] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:18.831071 16359 raft_consensus.cc:2258] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:18.833117 16360 ts_tablet_manager.cc:1905] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:18.834349 16091 tablet_service.cc:1467] Processing CreateTablet for tablet a57de77325c746609d7b4771e1599195 (DEFAULT_TABLE table=test-table [id=5f647cff800b4620b80019cfd4504d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.834609 16359 ts_tablet_manager.cc:1905] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:18.834661 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57de77325c746609d7b4771e1599195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.837283 16335 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Bootstrap starting.
I20260810 23:06:18.837461 16359 ts_tablet_manager.cc:1918] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:18.837575 16359 log.cc:1196] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/e13d65a898df4a339bef25d127f46867
I20260810 23:06:18.837873 16359 ts_tablet_manager.cc:1939] T e13d65a898df4a339bef25d127f46867 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting consensus metadata
I20260810 23:06:18.838320 16335 tablet_bootstrap.cc:654] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.839043 15789 catalog_manager.cc:4841] TS 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417): tablet e13d65a898df4a339bef25d127f46867 (table test-table [id=9f755f3ba9374bcab2b49bcc7b066293]) successfully deleted
I20260810 23:06:18.839285 16237 tablet_service.cc:1467] Processing CreateTablet for tablet a57de77325c746609d7b4771e1599195 (DEFAULT_TABLE table=test-table [id=5f647cff800b4620b80019cfd4504d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.839564 16237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57de77325c746609d7b4771e1599195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.841504 16360 ts_tablet_manager.cc:1918] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:18.841631 16360 log.cc:1196] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/e13d65a898df4a339bef25d127f46867
I20260810 23:06:18.842000 16360 ts_tablet_manager.cc:1939] T e13d65a898df4a339bef25d127f46867 P 40bab2a6360743f9bddcee440fc15446: Deleting consensus metadata
I20260810 23:06:18.842016 16333 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Bootstrap starting.
I20260810 23:06:18.842445 15944 tablet_service.cc:1467] Processing CreateTablet for tablet a57de77325c746609d7b4771e1599195 (DEFAULT_TABLE table=test-table [id=5f647cff800b4620b80019cfd4504d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:18.842685 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a57de77325c746609d7b4771e1599195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:18.842994 16333 tablet_bootstrap.cc:654] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.843350 15786 catalog_manager.cc:4841] TS 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573): tablet e13d65a898df4a339bef25d127f46867 (table test-table [id=9f755f3ba9374bcab2b49bcc7b066293]) successfully deleted
I20260810 23:06:18.844931 16334 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Bootstrap starting.
I20260810 23:06:18.845439 16335 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: No bootstrap required, opened a new log
I20260810 23:06:18.845536 16335 ts_tablet_manager.cc:1397] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 23:06:18.846053 16335 raft_consensus.cc:348] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.846179 16335 raft_consensus.cc:374] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.846210 16335 raft_consensus.cc:729] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bf63ed5f3d48f59e2889ffa2b26a5c, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.846293 16334 tablet_bootstrap.cc:654] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:18.846331 16335 consensus_queue.cc:260] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.846922 16335 ts_tablet_manager.cc:1428] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:18.847745 16361 ts_tablet_manager.cc:1905] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:18.847781 16334 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: No bootstrap required, opened a new log
I20260810 23:06:18.847867 16334 ts_tablet_manager.cc:1397] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:18.848312 16334 raft_consensus.cc:348] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.848397 16334 raft_consensus.cc:374] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.848417 16334 raft_consensus.cc:729] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bab2a6360743f9bddcee440fc15446, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.848541 16334 consensus_queue.cc:260] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.848795 16334 ts_tablet_manager.cc:1428] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:18.852095 16361 ts_tablet_manager.cc:1918] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:18.852231 16361 log.cc:1196] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/e13d65a898df4a339bef25d127f46867
I20260810 23:06:18.852571 16361 ts_tablet_manager.cc:1939] T e13d65a898df4a339bef25d127f46867 P 43c3723e1076480b88efea4c605a9e31: Deleting consensus metadata
I20260810 23:06:18.853976 15789 catalog_manager.cc:4841] TS 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841): tablet e13d65a898df4a339bef25d127f46867 (table test-table [id=9f755f3ba9374bcab2b49bcc7b066293]) successfully deleted
I20260810 23:06:18.854328 16333 tablet_bootstrap.cc:492] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: No bootstrap required, opened a new log
I20260810 23:06:18.854444 16333 ts_tablet_manager.cc:1397] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 23:06:18.854868 16333 raft_consensus.cc:348] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.854972 16333 raft_consensus.cc:374] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:18.854993 16333 raft_consensus.cc:729] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c3723e1076480b88efea4c605a9e31, State: Initialized, Role: FOLLOWER
I20260810 23:06:18.855144 16333 consensus_queue.cc:260] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:18.855880 16333 ts_tablet_manager.cc:1428] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:19.104893 16340 raft_consensus.cc:482] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:19.105077 16340 raft_consensus.cc:504] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.106274 16340 leader_election.cc:288] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573), 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417)
I20260810 23:06:19.111531 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.106763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:41585 (local address 127.14.75.2:35417)
0810 23:06:19.106898 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.106901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.107082 (+ 181us) server_negotiation.cc:407] Connection header received
0810 23:06:19.107163 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.107166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.107209 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.107298 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:19.107574 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.108333 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110155 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.110768 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110886 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.110955 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.111315 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.111391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:19.111531 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.106620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:45475 (local address 127.14.75.1:38573)
0810 23:06:19.106805 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.106810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.106831 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:06:19.107008 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.107011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.107076 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.107163 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:19.107508 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.108326 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110168 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.110795 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110892 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.110954 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.111315 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.111391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:06:19.111634 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.106573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35417 (local address 127.14.75.3:41585)
0810 23:06:19.106990 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.107003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.107085 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.107320 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.107325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.107344 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:19.107452 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.107461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.108436 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.108440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.110031 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.110042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.110927 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.111124 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.111205 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.111288 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.111423 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":261,"thread_start_us":108,"threads_started":1}
I20260810 23:06:19.112084 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.106480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:38573 (local address 127.14.75.3:45475)
0810 23:06:19.106754 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.106770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.106893 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.107195 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.107197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.107211 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:19.107377 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.107384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.108436 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.108440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.110029 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.110042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.110927 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.110932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.111145 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.111206 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.111288 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.111427 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":241,"thread_start_us":79,"threads_started":1}
I20260810 23:06:19.112197 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57de77325c746609d7b4771e1599195" candidate_uuid: "43c3723e1076480b88efea4c605a9e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" is_pre_election: true
I20260810 23:06:19.112465 16114 raft_consensus.cc:2454] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c3723e1076480b88efea4c605a9e31 in term 0.
I20260810 23:06:19.112864 16189 leader_election.cc:302] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c3723e1076480b88efea4c605a9e31, 74bf63ed5f3d48f59e2889ffa2b26a5c; no voters:
I20260810 23:06:19.113040 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57de77325c746609d7b4771e1599195" candidate_uuid: "43c3723e1076480b88efea4c605a9e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bab2a6360743f9bddcee440fc15446" is_pre_election: true
I20260810 23:06:19.113139 16340 raft_consensus.cc:2781] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:19.113209 16340 raft_consensus.cc:482] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:19.113200 15966 raft_consensus.cc:2454] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c3723e1076480b88efea4c605a9e31 in term 0.
I20260810 23:06:19.113265 16340 raft_consensus.cc:3037] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.114388 16340 raft_consensus.cc:504] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.114773 16340 leader_election.cc:288] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [CANDIDATE]: Term 1 election: Requested vote from peers 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573), 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417)
I20260810 23:06:19.115074 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57de77325c746609d7b4771e1599195" candidate_uuid: "43c3723e1076480b88efea4c605a9e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bab2a6360743f9bddcee440fc15446"
I20260810 23:06:19.115129 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a57de77325c746609d7b4771e1599195" candidate_uuid: "43c3723e1076480b88efea4c605a9e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c"
I20260810 23:06:19.115240 15966 raft_consensus.cc:3037] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.115247 16114 raft_consensus.cc:3037] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.116180 16114 raft_consensus.cc:2454] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c3723e1076480b88efea4c605a9e31 in term 1.
I20260810 23:06:19.116194 15966 raft_consensus.cc:2454] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c3723e1076480b88efea4c605a9e31 in term 1.
I20260810 23:06:19.116592 16186 leader_election.cc:302] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bab2a6360743f9bddcee440fc15446, 43c3723e1076480b88efea4c605a9e31; no voters:
I20260810 23:06:19.116827 16340 raft_consensus.cc:2781] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:19.117072 16340 raft_consensus.cc:686] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 1 LEADER]: Becoming Leader. State: Replica: 43c3723e1076480b88efea4c605a9e31, State: Running, Role: LEADER
I20260810 23:06:19.117358 16340 consensus_queue.cc:237] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.120208 15801 catalog_manager.cc:5495] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43c3723e1076480b88efea4c605a9e31 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "43c3723e1076480b88efea4c605a9e31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } health_report { overall_health: HEALTHY } } }
Aug 10 23:06:19 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403177, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:19.167665 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.159291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.0.0.1:52838)
0810 23:06:19.159608 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.159625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.159745 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.160016 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.160019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.160454 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:19.160560 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.160566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.161326 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.161332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.161929 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.161938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.162397 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.162400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.162480 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.163101 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:19.163114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:19.164625 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:19.166260 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:19.166269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:19.166274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:19.166510 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:19.166754 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:19.166757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:19.166758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:19.166837 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:19.167165 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:19.167170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:19.167274 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.167363 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.167486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260810 23:06:19.167683 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.159397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.14.75.62:41569)
0810 23:06:19.159673 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.159685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.159715 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:06:19.159865 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.159868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.159914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.160001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:19.160678 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.161223 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.162047 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.162279 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.162381 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.164761 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:19.164765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:19.164767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:19.164788 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:19.166159 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.166598 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.166600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.166601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.166677 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.166909 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.166911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.166913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.167080 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:19.167152 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.167416 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.167548 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260810 23:06:19.171561 15801 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:06:19.172331 15801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:19.177090 16091 tablet_service.cc:1467] Processing CreateTablet for tablet 6295cf64b94b4df684dbd53ae82a6655 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a607485113094e04a1728364338a8bc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:19.177063 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 6295cf64b94b4df684dbd53ae82a6655 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a607485113094e04a1728364338a8bc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:19.177134 16237 tablet_service.cc:1467] Processing CreateTablet for tablet 6295cf64b94b4df684dbd53ae82a6655 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a607485113094e04a1728364338a8bc0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:19.177510 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6295cf64b94b4df684dbd53ae82a6655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:19.177510 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6295cf64b94b4df684dbd53ae82a6655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:19.177608 16237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6295cf64b94b4df684dbd53ae82a6655. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:19.180320 16333 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: Bootstrap starting.
I20260810 23:06:19.180320 16335 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Bootstrap starting.
I20260810 23:06:19.180341 16334 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: Bootstrap starting.
I20260810 23:06:19.181250 16334 tablet_bootstrap.cc:654] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:19.181308 16333 tablet_bootstrap.cc:654] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:19.181308 16335 tablet_bootstrap.cc:654] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:19.182556 16334 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: No bootstrap required, opened a new log
I20260810 23:06:19.182643 16334 ts_tablet_manager.cc:1397] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:19.182677 16333 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: No bootstrap required, opened a new log
I20260810 23:06:19.182745 16333 ts_tablet_manager.cc:1397] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:19.182912 16335 tablet_bootstrap.cc:492] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: No bootstrap required, opened a new log
I20260810 23:06:19.182974 16335 ts_tablet_manager.cc:1397] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:19.183118 16334 raft_consensus.cc:348] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183122 16333 raft_consensus.cc:348] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183228 16334 raft_consensus.cc:374] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:19.183239 16333 raft_consensus.cc:374] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:19.183266 16334 raft_consensus.cc:729] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bab2a6360743f9bddcee440fc15446, State: Initialized, Role: FOLLOWER
I20260810 23:06:19.183276 16333 raft_consensus.cc:729] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c3723e1076480b88efea4c605a9e31, State: Initialized, Role: FOLLOWER
I20260810 23:06:19.183272 16335 raft_consensus.cc:348] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183353 16335 raft_consensus.cc:374] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:19.183413 16335 raft_consensus.cc:729] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bf63ed5f3d48f59e2889ffa2b26a5c, State: Initialized, Role: FOLLOWER
I20260810 23:06:19.183395 16333 consensus_queue.cc:260] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183395 16334 consensus_queue.cc:260] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183552 16335 consensus_queue.cc:260] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.183703 16334 ts_tablet_manager.cc:1428] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:19.183825 16335 ts_tablet_manager.cc:1428] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:19.183847 16354 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.183979 16354 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.184078 16333 ts_tablet_manager.cc:1428] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:19.184247 16340 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.184369 16340 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.184476 16341 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.184597 16341 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.507849 16341 raft_consensus.cc:482] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:19.507998 16341 raft_consensus.cc:504] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.509460 16341 leader_election.cc:288] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417), 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841)
I20260810 23:06:19.513752 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.509905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:47155 (local address 127.14.75.2:35417)
0810 23:06:19.510029 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.510032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.510132 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:06:19.510373 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.510377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.510439 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.510499 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:19.510934 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.511493 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.512724 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.513117 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.513233 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.513300 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.513531 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.513608 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:06:19.513777 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.509791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35417 (local address 127.14.75.1:47155)
0810 23:06:19.510053 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.510067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.510194 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.510660 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.510663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.510684 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:19.510813 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.510820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.511613 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.511620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.512574 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.512585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.513246 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.513250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.513340 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.513368 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.513511 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.513597 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":47,"thread_start_us":100,"threads_started":1}
I20260810 23:06:19.513788 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.510203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36841 (local address 127.14.75.1:52587)
0810 23:06:19.510446 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.510462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.510570 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.510816 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.510817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.510827 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:19.510941 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.510947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.511678 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.511680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.512564 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.512573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.513268 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.513271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.513377 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.513393 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.513504 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.513580 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260810 23:06:19.513924 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.510333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:52587 (local address 127.14.75.3:36841)
0810 23:06:19.510591 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.510598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.510614 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:19.510665 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.510668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.510722 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.510813 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:19.511030 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.511538 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.512715 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.513132 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.513255 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.513320 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.513681 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.513769 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260810 23:06:19.514422 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6295cf64b94b4df684dbd53ae82a6655" candidate_uuid: "40bab2a6360743f9bddcee440fc15446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" is_pre_election: true
I20260810 23:06:19.514424 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6295cf64b94b4df684dbd53ae82a6655" candidate_uuid: "40bab2a6360743f9bddcee440fc15446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c3723e1076480b88efea4c605a9e31" is_pre_election: true
I20260810 23:06:19.514585 16114 raft_consensus.cc:2454] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bab2a6360743f9bddcee440fc15446 in term 0.
I20260810 23:06:19.514590 16259 raft_consensus.cc:2454] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bab2a6360743f9bddcee440fc15446 in term 0.
I20260810 23:06:19.514947 15897 leader_election.cc:302] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bab2a6360743f9bddcee440fc15446, 74bf63ed5f3d48f59e2889ffa2b26a5c; no voters:
I20260810 23:06:19.515240 16341 raft_consensus.cc:2781] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:19.515298 16341 raft_consensus.cc:482] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:19.515324 16341 raft_consensus.cc:3037] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.516412 16341 raft_consensus.cc:504] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.516757 16341 leader_election.cc:288] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [CANDIDATE]: Term 1 election: Requested vote from peers 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417), 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841)
I20260810 23:06:19.517141 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6295cf64b94b4df684dbd53ae82a6655" candidate_uuid: "40bab2a6360743f9bddcee440fc15446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c"
I20260810 23:06:19.517177 16259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6295cf64b94b4df684dbd53ae82a6655" candidate_uuid: "40bab2a6360743f9bddcee440fc15446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c3723e1076480b88efea4c605a9e31"
I20260810 23:06:19.517287 16114 raft_consensus.cc:3037] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.517302 16259 raft_consensus.cc:3037] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:19.518234 16259 raft_consensus.cc:2454] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bab2a6360743f9bddcee440fc15446 in term 1.
I20260810 23:06:19.518327 16114 raft_consensus.cc:2454] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bab2a6360743f9bddcee440fc15446 in term 1.
I20260810 23:06:19.518532 15897 leader_election.cc:302] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bab2a6360743f9bddcee440fc15446, 43c3723e1076480b88efea4c605a9e31; no voters:
I20260810 23:06:19.518730 16341 raft_consensus.cc:2781] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:19.518935 16341 raft_consensus.cc:686] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [term 1 LEADER]: Becoming Leader. State: Replica: 40bab2a6360743f9bddcee440fc15446, State: Running, Role: LEADER
I20260810 23:06:19.519207 16341 consensus_queue.cc:237] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } }
I20260810 23:06:19.520677 16383 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: TxnStatusTablet state changed. Reason: New leader 40bab2a6360743f9bddcee440fc15446. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.520793 16383 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:19.520781 16340 consensus_queue.cc:1035] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:19.521355 16385 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:06:19.521621 16385 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:06:19.522253 15802 catalog_manager.cc:5495] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40bab2a6360743f9bddcee440fc15446 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:19.526687 16367 consensus_queue.cc:1035] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:19.556872 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.552821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:38573 (local address 127.0.0.1:33426)
0810 23:06:19.553005 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.553027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.553117 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.553483 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.553485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.553495 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:19.553591 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.553596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.554572 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.554574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.555173 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.555180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.555741 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.555744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.555809 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.555855 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.556618 (+ 763us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.556629 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.556675 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.556754 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:06:19.556910 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.552930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.14.75.1:38573)
0810 23:06:19.553080 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.553086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.553103 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:19.553267 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.553270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.553338 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.553467 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:19.553695 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.554472 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.555281 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.555621 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.555735 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.555991 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.556521 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.556628 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.556690 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.556750 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:19.559927 16259 raft_consensus.cc:1261] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Refusing update from remote peer 40bab2a6360743f9bddcee440fc15446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:19.559944 16114 raft_consensus.cc:1261] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Refusing update from remote peer 40bab2a6360743f9bddcee440fc15446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:19.560518 16341 consensus_queue.cc:1035] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:19.560796 16383 consensus_queue.cc:1035] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:19.562256 16367 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: TxnStatusTablet state changed. Reason: New leader 40bab2a6360743f9bddcee440fc15446. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.562393 16367 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.563323 16394 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: TxnStatusTablet state changed. Reason: New leader 40bab2a6360743f9bddcee440fc15446. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.563512 16394 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.565048 16383 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.565165 16383 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:19.565292 16383 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.565373 16383 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 40bab2a6360743f9bddcee440fc15446: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:19.565742 16367 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.565862 16367 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 43c3723e1076480b88efea4c605a9e31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.567826 16394 tablet_replica.cc:439] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40bab2a6360743f9bddcee440fc15446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bab2a6360743f9bddcee440fc15446" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 38573 } } peers { permanent_uuid: "74bf63ed5f3d48f59e2889ffa2b26a5c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35417 } } peers { permanent_uuid: "43c3723e1076480b88efea4c605a9e31" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36841 } } }
I20260810 23:06:19.567941 16394 tablet_replica.cc:442] T 6295cf64b94b4df684dbd53ae82a6655 P 74bf63ed5f3d48f59e2889ffa2b26a5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:19.576339 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.571890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.14.75.3:36841)
0810 23:06:19.572017 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.572020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.572128 (+ 108us) server_negotiation.cc:407] Connection header received
0810 23:06:19.572280 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.572282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.572335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.572425 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:19.572742 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.573555 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.574792 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.575007 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.575089 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.575368 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.575937 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.576027 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.576124 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.576177 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:19.587759 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.583687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:38573 (local address 127.14.75.3:54533)
0810 23:06:19.583999 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.584013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.584120 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.584384 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.584386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.584392 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:19.584826 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.584832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.585865 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.585867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.586486 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.586494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.587060 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.587063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.587134 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.587181 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.587481 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.587486 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.587537 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.587602 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":110,"threads_started":1}
I20260810 23:06:19.587764 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.583821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:54533 (local address 127.14.75.1:38573)
0810 23:06:19.584032 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.584037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.584086 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:06:19.584226 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.584230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.584284 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.584359 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:19.584938 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.585761 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.586601 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.586942 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.587040 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.587310 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.587399 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.587471 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.587570 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.587623 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 23:06:19.616528 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.612607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:33967 (local address 127.14.75.3:36841)
0810 23:06:19.612743 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.612748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.612938 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:06:19.613123 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.613126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.613179 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.613253 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:19.613581 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.614257 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.615438 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.615674 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.615784 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.616087 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.616166 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.616241 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.616328 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.616379 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:06:19.616560 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.612523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36841 (local address 127.14.75.1:33967)
0810 23:06:19.612858 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:19.612871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:19.612990 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:19.613275 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:19.613278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:19.613285 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:19.613468 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.613476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.614394 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.614397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.615321 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:19.615331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.615785 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.615789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:19.615881 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.615934 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:19.616245 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:19.616254 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:06:19.616319 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:06:19.616399 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260810 23:06:19.640049 15802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:19.640224 15802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:19.642395 15802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00a95867213647d380cb52d06aac8810: Sending DeleteTablet for 3 replicas of tablet a57de77325c746609d7b4771e1599195
I20260810 23:06:19.643266 16091 tablet_service.cc:1514] Processing DeleteTablet for tablet a57de77325c746609d7b4771e1599195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:19 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:34768
I20260810 23:06:19.643477 16237 tablet_service.cc:1514] Processing DeleteTablet for tablet a57de77325c746609d7b4771e1599195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:19 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260810 23:06:19.643826 16412 tablet_replica.cc:330] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: stopping tablet replica
I20260810 23:06:19.643983 16412 raft_consensus.cc:2229] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:19.644150 16412 raft_consensus.cc:2258] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:19.644932 16412 ts_tablet_manager.cc:1905] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:19.645277 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet a57de77325c746609d7b4771e1599195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:19 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:33418
I20260810 23:06:19.645722 16413 tablet_replica.cc:330] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: stopping tablet replica
I20260810 23:06:19.645852 16413 raft_consensus.cc:2229] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:19.646157 16413 raft_consensus.cc:2258] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:19.647909 16413 ts_tablet_manager.cc:1905] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:19.648674 16412 ts_tablet_manager.cc:1918] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:19.648768 16412 log.cc:1196] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/a57de77325c746609d7b4771e1599195
I20260810 23:06:19.649065 16412 ts_tablet_manager.cc:1939] T a57de77325c746609d7b4771e1599195 P 74bf63ed5f3d48f59e2889ffa2b26a5c: Deleting consensus metadata
I20260810 23:06:19.649924 15789 catalog_manager.cc:4841] TS 74bf63ed5f3d48f59e2889ffa2b26a5c (127.14.75.2:35417): tablet a57de77325c746609d7b4771e1599195 (table test-table [id=5f647cff800b4620b80019cfd4504d38]) successfully deleted
I20260810 23:06:19.650915 16413 ts_tablet_manager.cc:1918] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:19.650985 16413 log.cc:1196] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/a57de77325c746609d7b4771e1599195
I20260810 23:06:19.651285 16413 ts_tablet_manager.cc:1939] T a57de77325c746609d7b4771e1599195 P 43c3723e1076480b88efea4c605a9e31: Deleting consensus metadata
I20260810 23:06:19.652050 15789 catalog_manager.cc:4841] TS 43c3723e1076480b88efea4c605a9e31 (127.14.75.3:36841): tablet a57de77325c746609d7b4771e1599195 (table test-table [id=5f647cff800b4620b80019cfd4504d38]) successfully deleted
I20260810 23:06:19.653600 16415 tablet_replica.cc:330] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: stopping tablet replica
I20260810 23:06:19.653755 16415 raft_consensus.cc:2229] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:19.653889 16415 raft_consensus.cc:2258] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 23:06:19 dist-test-slave-srgv krb5kdc[15756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403178, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:19.662034 16415 ts_tablet_manager.cc:1905] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:19.663373 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.646683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.14.75.1:38573)
0810 23:06:19.646803 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.646808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.652197 (+ 5389us) server_negotiation.cc:407] Connection header received
0810 23:06:19.652417 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.652421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.652479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.652572 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:19.653268 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.654014 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.654950 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.655187 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.655300 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.658744 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:19.658755 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:19.658768 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:19.658799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:19.660625 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.661122 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.661128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.661129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.661228 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.661595 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.661599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.661600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.661870 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:19.662720 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.662807 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.662916 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:06:19.663836 16008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33428
I20260810 23:06:19.665477 16415 ts_tablet_manager.cc:1918] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:19.665573 16415 log.cc:1196] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/a57de77325c746609d7b4771e1599195
I20260810 23:06:19.665867 16415 ts_tablet_manager.cc:1939] T a57de77325c746609d7b4771e1599195 P 40bab2a6360743f9bddcee440fc15446: Deleting consensus metadata
I20260810 23:06:19.666637 15786 catalog_manager.cc:4841] TS 40bab2a6360743f9bddcee440fc15446 (127.14.75.1:38573): tablet a57de77325c746609d7b4771e1599195 (table test-table [id=5f647cff800b4620b80019cfd4504d38]) successfully deleted
I20260810 23:06:19.674974 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:19.665754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.14.75.62:41569)
0810 23:06:19.666630 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:19.666635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:19.666653 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:06:19.666700 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:19.666703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:19.666752 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:19.666845 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:19.667510 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.668061 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.668846 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:19.669099 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:19.669207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:19.670829 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:19.670833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:19.670835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:19.670870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:19.672395 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.672900 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.672905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.672909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.673047 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:19.673335 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:19.673339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:19.673341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:19.673610 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:19.673754 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:19.674645 (+ 891us) server_negotiation.cc:299] Negotiation successful
0810 23:06:19.674796 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774}
W20260810 23:06:19.676223 15802 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52848
I20260810 23:06:19.678946 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15878
I20260810 23:06:19.689137 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16024
I20260810 23:06:19.697224 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16170
I20260810 23:06:19.705039 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 15769
2026-08-10T23:06:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3600 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:19 dist-test-slave-srgv krb5kdc[16421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:19 dist-test-slave-srgv krb5kdc[16421](info): set up 2 sockets
Aug 10 23:06:19 dist-test-slave-srgv krb5kdc[16421](info): commencing operation
krb5kdc: starting...
W20260810 23:06:20.762765 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:20 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:20.908519 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43501
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:43501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:21.022049 16435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:21.022329 16435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:21.022372 16435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:21.026510 16435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:21.026577 16435 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:06:21.026594 16435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:21.026613 16435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:21.026629 16435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:21.026678 16435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:21.032164 16435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:43501
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:21.033272 16435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:21.034332 16435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:21.042928 16440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.042956 16441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.042948 16443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:21.043219 16435 server_base.cc:1034] running on GCE node
I20260810 23:06:21.043960 16435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:21.045192 16435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:21.046366 16435 hybrid_clock.cc:648] HybridClock initialized: now 1786403181046345 us; error 33 us; skew 500 ppm
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:21.049197 16435 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:21.050273 16435 webserver.cc:460] Webserver started at http://127.14.75.62:32837/ using document root <none> and password file <none>
I20260810 23:06:21.050853 16435 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:21.050943 16435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:21.051172 16435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:21.053335 16435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "a2a52e3cfdd9422aade931788bdf87b9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "4523f2b0992f112f913bf3a4de2cef70"
server_key_iv: "c8374c2e0382e2945ba92387f355ec06"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.053982 16435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "a2a52e3cfdd9422aade931788bdf87b9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "4523f2b0992f112f913bf3a4de2cef70"
server_key_iv: "c8374c2e0382e2945ba92387f355ec06"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.057881 16435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:21.060322 16450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.061544 16435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:21.061689 16435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "a2a52e3cfdd9422aade931788bdf87b9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "4523f2b0992f112f913bf3a4de2cef70"
server_key_iv: "c8374c2e0382e2945ba92387f355ec06"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.061806 16435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:21.086076 16435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:21.089130 16435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:21.089346 16435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:21.097430 16435 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:43501
I20260810 23:06:21.097465 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:43501 every 8 connection(s)
I20260810 23:06:21.098479 16435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:21.101219 16517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:21.105706 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16435
I20260810 23:06:21.105837 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:21.106107 14636 external_mini_cluster.cc:1428] Setting key 6f09d89ab3053b05bb11d98ef406c55a
I20260810 23:06:21.107182 16517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Bootstrap starting.
I20260810 23:06:21.110018 16517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:21.110873 16517 log.cc:824] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:21.113005 16517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: No bootstrap required, opened a new log
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:21.117574 16517 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } }
I20260810 23:06:21.117916 16517 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:21.117988 16517 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a52e3cfdd9422aade931788bdf87b9, State: Initialized, Role: FOLLOWER
I20260810 23:06:21.118501 16517 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } }
I20260810 23:06:21.118669 16517 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:21.118762 16517 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:21.118880 16517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:21.120146 16517 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } }
I20260810 23:06:21.120553 16517 leader_election.cc:302] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a52e3cfdd9422aade931788bdf87b9; no voters:
I20260810 23:06:21.120941 16517 leader_election.cc:288] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:21.121129 16522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:21.121410 16522 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [term 1 LEADER]: Becoming Leader. State: Replica: a2a52e3cfdd9422aade931788bdf87b9, State: Running, Role: LEADER
I20260810 23:06:21.121768 16522 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } }
I20260810 23:06:21.122296 16517 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:21.122311 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.107553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.14.75.62:43501)
0810 23:06:21.108020 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.108039 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.108081 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:06:21.108969 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.108993 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.109437 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.109797 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.110700 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.112073 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.112911 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.113647 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.113807 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.116575 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.116581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.116598 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.116634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.118708 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.119312 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.119317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.119323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.119458 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.119767 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.119773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.119776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.120347 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.120552 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.120815 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.121325 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":131,"threads_started":1}
I20260810 23:06:21.123018 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a52e3cfdd9422aade931788bdf87b9. Latest consensus state: current_term: 1 leader_uuid: "a2a52e3cfdd9422aade931788bdf87b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } } }
I20260810 23:06:21.123157 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:21.123559 16524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a52e3cfdd9422aade931788bdf87b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a52e3cfdd9422aade931788bdf87b9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43501 } } }
I20260810 23:06:21.123620 16531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:21.123697 16524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:21.126832 16531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:21.133375 16531 catalog_manager.cc:1347] Generated new cluster ID: 2e8e3db96bc048db9ed09061715fa315
I20260810 23:06:21.133498 16531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:21.146709 16531 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:21.148042 16531 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:21.159230 16531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Generated new TSK 0
I20260810 23:06:21.160053 16531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:21.196736 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43501
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:21.324642 16544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:21.324920 16544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:21.324972 16544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:21.329142 16544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:21.329216 16544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:21.329257 16544 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:06:21.329272 16544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:21.329303 16544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:21.334772 16544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:21.335861 16544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:21.336858 16544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:21.346901 16544 server_base.cc:1034] running on GCE node
W20260810 23:06:21.346844 16552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.346844 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.346845 16549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:21.347550 16544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:21.348222 16544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:21.349444 16544 hybrid_clock.cc:648] HybridClock initialized: now 1786403181349397 us; error 30 us; skew 500 ppm
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:21.352185 16544 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:21.353078 16544 webserver.cc:460] Webserver started at http://127.14.75.1:34405/ using document root <none> and password file <none>
I20260810 23:06:21.353713 16544 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:21.353804 16544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:21.354025 16544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:21.356113 16544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "f9eab29d1bc34310a4205519ca2dde34"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "fa0455bb4092e9cde5b4604df66454cf"
server_key_iv: "3c95eaabee3a9965f088cca4cadad0c6"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.356774 16544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "f9eab29d1bc34310a4205519ca2dde34"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "fa0455bb4092e9cde5b4604df66454cf"
server_key_iv: "3c95eaabee3a9965f088cca4cadad0c6"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.360651 16544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:21.363240 16560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.364552 16544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:21.364699 16544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "f9eab29d1bc34310a4205519ca2dde34"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "fa0455bb4092e9cde5b4604df66454cf"
server_key_iv: "3c95eaabee3a9965f088cca4cadad0c6"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.364828 16544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:21.392089 16544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:21.395429 16544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:21.395643 16544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:21.397253 16544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:21.397317 16544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.397367 16544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:21.397404 16544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:21.408907 16544 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:39597
I20260810 23:06:21.408938 16681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:39597 every 8 connection(s)
I20260810 23:06:21.410137 16544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:21.412026 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.398182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:41851 (local address 127.14.75.62:43501)
0810 23:06:21.398375 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.398382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.399305 (+ 923us) server_negotiation.cc:407] Connection header received
0810 23:06:21.400726 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.400732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.400795 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.400904 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.402757 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.403373 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.404638 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.404936 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.405059 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.407863 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.407866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.407868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.407896 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.409807 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.410397 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.410402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.410404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.410519 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.410955 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.410959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.410960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.411197 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.411291 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.411786 (+ 495us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.411904 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:06:21.412842 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.398619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.1:41851)
0810 23:06:21.399107 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.399168 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.400386 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.401138 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.401149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.401755 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.402540 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.402578 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.403529 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.403538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.404502 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.404514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.405118 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.405122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.405354 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.406078 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.406094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.407666 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.409934 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.409939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.409952 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.410286 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.410650 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.410655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.410657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.410830 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.411320 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.411326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.411571 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.411735 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.412003 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
I20260810 23:06:21.414192 16682 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43501
I20260810 23:06:21.414369 16682 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:21.414907 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16544
I20260810 23:06:21.415040 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:21.415019 16682 heartbeater.cc:511] Master 127.14.75.62:43501 requested a full tablet report, sending...
I20260810 23:06:21.415354 14636 external_mini_cluster.cc:1428] Setting key d02e7f916ab8c3e7cf9e4a67dc4e7ee5
I20260810 23:06:21.416745 16468 ts_manager.cc:194] Registered new tserver with Master: f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597)
I20260810 23:06:21.418111 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:41851
I20260810 23:06:21.425531 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.416755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:34247 (local address 127.14.75.62:43501)
0810 23:06:21.416886 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.416891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.417142 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:06:21.417241 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.417244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.417288 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.417365 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.417952 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.418650 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.419966 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.420298 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.420409 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.421846 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.421849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.421851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.421875 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.423549 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.424188 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.424191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.424192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.424277 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.424605 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.424607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.424608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.424792 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.424920 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.425235 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.425341 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:06:21.425649 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.416780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.1:34247)
0810 23:06:21.417053 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.417066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.417151 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.417387 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.417390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.417669 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.417824 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.417833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.418809 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.418813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.419826 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.419840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.420436 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.420443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.420563 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.421102 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.421122 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.421690 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.423677 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.423684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.423689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.424052 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.424378 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.424384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.424387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.424509 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.424927 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.424930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.425057 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.425180 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.425376 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:21.446712 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43501
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:21.561348 16692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:21.561802 16692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:21.561908 16692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:21.566578 16692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:21.566644 16692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:21.566712 16692 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:06:21.566743 16692 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:21.566798 16692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:21.572446 16692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:21.573702 16692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:21.574857 16692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:21.584841 16701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.584888 16698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.584846 16699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:21.585064 16692 server_base.cc:1034] running on GCE node
I20260810 23:06:21.585656 16692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:21.586341 16692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:21.587527 16692 hybrid_clock.cc:648] HybridClock initialized: now 1786403181587514 us; error 33 us; skew 500 ppm
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:21.590389 16692 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:21.591308 16692 webserver.cc:460] Webserver started at http://127.14.75.2:36085/ using document root <none> and password file <none>
I20260810 23:06:21.591904 16692 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:21.591995 16692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:21.592243 16692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:21.594650 16692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "a01e2d092baf4d809c66ee08b242a676"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "03df1c29e34fcd60ed10d1f9aa772ab0"
server_key_iv: "373f01c13cc1213fa831c974b82c69c1"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.595366 16692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "a01e2d092baf4d809c66ee08b242a676"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "03df1c29e34fcd60ed10d1f9aa772ab0"
server_key_iv: "373f01c13cc1213fa831c974b82c69c1"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.599402 16692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:21.601902 16708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.603075 16692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:21.603215 16692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "a01e2d092baf4d809c66ee08b242a676"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "03df1c29e34fcd60ed10d1f9aa772ab0"
server_key_iv: "373f01c13cc1213fa831c974b82c69c1"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.603338 16692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:21.616573 16692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:21.620138 16692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:21.620358 16692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:21.621928 16692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:21.621982 16692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.622044 16692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:21.622076 16692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.632891 16692 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:44961
I20260810 23:06:21.632962 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:44961 every 8 connection(s)
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:21.634058 16692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:21.637640 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.622927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:38773 (local address 127.14.75.62:43501)
0810 23:06:21.623290 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.623293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.624160 (+ 867us) server_negotiation.cc:407] Connection header received
0810 23:06:21.625385 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.625389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.625497 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.625600 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.627471 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.628415 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.630160 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.630548 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.630690 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.634004 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.634011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.634017 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.634052 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.635777 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.636269 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.636271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.636272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.636350 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.636634 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.636636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.636638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.636817 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.636891 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.637344 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.637516 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260810 23:06:21.638489 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.623516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.2:38773)
0810 23:06:21.623999 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.624056 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.625135 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.625948 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.625959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.626493 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.627259 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.627284 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.628557 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.628563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.630025 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.630035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.630738 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.630742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.630959 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.631775 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.631805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.633841 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.635881 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.635885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.635897 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.636176 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.636426 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.636428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.636429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.636558 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.636900 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.636905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.637134 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.637299 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.637631 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260810 23:06:21.639770 16829 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43501
I20260810 23:06:21.639925 16829 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:21.640417 16829 heartbeater.cc:511] Master 127.14.75.62:43501 requested a full tablet report, sending...
I20260810 23:06:21.641517 16469 ts_manager.cc:194] Registered new tserver with Master: a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961)
I20260810 23:06:21.642197 16469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:38773
I20260810 23:06:21.644191 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16692
I20260810 23:06:21.644285 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:21.644552 14636 external_mini_cluster.cc:1428] Setting key 29f53603c965e74ac73afbd3805d009a
I20260810 23:06:21.649124 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.642002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:57873 (local address 127.14.75.62:43501)
0810 23:06:21.642137 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.642141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.642378 (+ 237us) server_negotiation.cc:407] Connection header received
0810 23:06:21.642510 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.642513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.642560 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.642640 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.643087 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.643595 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.644468 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.644744 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.644851 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.645999 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.646003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.646006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.646029 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.647423 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.647856 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.647861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.647863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.647947 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.648246 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.648250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.648252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.648476 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.648584 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.648849 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.648965 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:21.649138 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.641924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.2:57873)
0810 23:06:21.642305 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.642321 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.642416 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.642656 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.642659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.642882 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.642979 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.642984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.643723 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.643727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.644345 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.644356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.644856 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.644859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.644952 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.645454 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.645467 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.645863 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.647540 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.647544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.647547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.647759 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.648036 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.648040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.648042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.648132 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.648573 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.648576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.648685 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.648791 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.648952 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:21.670915 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43501
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:21.787115 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:21.787402 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:21.787489 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:21.791795 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:21.791869 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:21.791913 16841 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:06:21.791965 16841 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:21.792032 16841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:21.797574 16841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:21.798674 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:21.799708 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:21.809504 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.809546 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:21.809520 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:21.809610 16841 server_base.cc:1034] running on GCE node
I20260810 23:06:21.810210 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:21.810851 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:21.812012 16841 hybrid_clock.cc:648] HybridClock initialized: now 1786403181811997 us; error 28 us; skew 500 ppm
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:21.814961 16841 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:21.815938 16841 webserver.cc:460] Webserver started at http://127.14.75.3:37957/ using document root <none> and password file <none>
I20260810 23:06:21.816510 16841 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:21.816573 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:21.816826 16841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:21.819025 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "c3374fef4b12417488e9cb0c4704d4f9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "a0c325b272dd65ae1032a1d65583ad3f"
server_key_iv: "cb21bb74dcd48c3bf23285fa4af82e34"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.819659 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "c3374fef4b12417488e9cb0c4704d4f9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "a0c325b272dd65ae1032a1d65583ad3f"
server_key_iv: "cb21bb74dcd48c3bf23285fa4af82e34"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.823832 16841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:21.826402 16856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.827647 16841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:21.827769 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "c3374fef4b12417488e9cb0c4704d4f9"
format_stamp: "Formatted at 2026-08-10 23:06:21 on dist-test-slave-srgv"
server_key: "a0c325b272dd65ae1032a1d65583ad3f"
server_key_iv: "cb21bb74dcd48c3bf23285fa4af82e34"
server_key_version: "encryptionkey@0"
I20260810 23:06:21.827883 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:21.840243 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:21.843745 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:21.843952 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:21.845538 16841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:21.845597 16841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:21.845661 16841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:21.845695 16841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:21.857360 16841 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45915
I20260810 23:06:21.857463 16978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45915 every 8 connection(s)
I20260810 23:06:21.858501 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:21.860091 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.846299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:34631 (local address 127.14.75.62:43501)
0810 23:06:21.846456 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.846461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.847377 (+ 916us) server_negotiation.cc:407] Connection header received
0810 23:06:21.848385 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.848389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.848453 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.848516 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.850020 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.850651 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.851870 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.852092 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.852208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.855538 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.855541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.855543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.855573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.857596 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.858122 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.858128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.858133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.858236 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.858586 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.858590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.858592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.858852 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.858981 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.859842 (+ 861us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.859962 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:06:21.860770 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.846736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.3:34631)
0810 23:06:21.847209 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.847260 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.848126 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.848872 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.848878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.849234 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.849874 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.849890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.850798 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.850801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.851744 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.851754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.852261 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.852266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.852434 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.853109 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.853126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.855332 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.857718 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.857724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.857742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.858003 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.858337 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.858339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.858341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.858476 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.859062 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.859070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.859407 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.859623 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.859899 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
I20260810 23:06:21.862150 16979 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43501
I20260810 23:06:21.862308 16979 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:21.862825 16979 heartbeater.cc:511] Master 127.14.75.62:43501 requested a full tablet report, sending...
I20260810 23:06:21.863850 16468 ts_manager.cc:194] Registered new tserver with Master: c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:21.864504 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:34631
I20260810 23:06:21.867950 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 16841
I20260810 23:06:21.868088 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:21.868376 14636 external_mini_cluster.cc:1428] Setting key 8ae90f9858f74f843a188bfc7fa98715
I20260810 23:06:21.870761 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:21.872246 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.864506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:57287 (local address 127.14.75.62:43501)
0810 23:06:21.864641 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.864659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.864923 (+ 264us) server_negotiation.cc:407] Connection header received
0810 23:06:21.865016 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.865019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.865069 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.865155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.865803 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.866350 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.867246 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.867451 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.867529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.868872 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.868875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.868878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.868899 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.870650 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.871111 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.871114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.871115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.871191 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.871482 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.871484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.871485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.871647 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.871716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.872039 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.872135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:21.872457 16985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.864446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.14.75.3:57287)
0810 23:06:21.864830 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.864846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.864949 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.865287 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.865290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.865565 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:21.865692 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.865700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.866481 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.866484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.867138 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.867154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.867546 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.867548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.867640 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.868261 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:21.868274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:21.868727 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:21.870763 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.870766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.870768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.871020 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.871283 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:21.871286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:21.871289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:21.871400 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:21.871738 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:21.871744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:21.871862 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.871980 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.872174 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":100,"threads_started":1}
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:06:21 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:21.897161 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.889522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.14.75.62:43501)
0810 23:06:21.889645 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.889649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.889663 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:21.889708 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.889710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.889758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.889836 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:21.890491 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.891012 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.891780 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.891970 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.892040 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.894443 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:21.894446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:21.894448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:21.894470 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:21.895836 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.896195 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.896198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.896199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.896270 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:21.896486 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:21.896488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:21.896489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:21.896657 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:21.896759 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.896944 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.897038 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:06:21.900980 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:21.904166 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:21.918850 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.912997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.0.0.1:58912)
0810 23:06:21.914220 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.914231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.914338 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.914801 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.914804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.914820 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:21.914942 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.914950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.916271 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.916274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.917523 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.917533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.918304 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.918307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.918422 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.918462 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.918578 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.918694 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":51,"thread_start_us":87,"threads_started":1}
I20260810 23:06:21.918862 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.913532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.14.75.3:45915)
0810 23:06:21.914142 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.914165 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.914426 (+ 261us) server_negotiation.cc:407] Connection header received
0810 23:06:21.914523 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.914528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.914678 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.914823 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:21.915069 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.916151 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.917632 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.918177 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.918305 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.918476 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.918608 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.918700 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":123,"threads_started":1}
I20260810 23:06:21.919569 16999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.912781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:44961 (local address 127.0.0.1:45858)
0810 23:06:21.913872 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.913887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.913978 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.915559 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.915562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.915580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:21.915691 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.915701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.917040 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.917043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.918177 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.918188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.919287 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.919290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.919364 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.919381 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.919429 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.919476 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":109,"threads_started":1}
I20260810 23:06:21.919708 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.912926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.14.75.2:44961)
0810 23:06:21.915115 (+ 2189us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.915127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.915143 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:21.915251 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.915256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.915411 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.915580 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:21.915811 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.916909 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.918334 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.919177 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.919291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.919436 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.919481 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.919560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2082,"thread_start_us":114,"threads_started":1}
I20260810 23:06:21.921823 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.912607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39597 (local address 127.0.0.1:42592)
0810 23:06:21.913483 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:21.913552 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:21.913679 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:21.914423 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:21.914432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:21.914456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:21.914593 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.914599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.916019 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.916022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.917553 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:21.917564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.918852 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.918855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:21.921519 (+ 2664us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.921538 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:06:21.921617 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:06:21.921688 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":70,"thread_start_us":103,"threads_started":1}
I20260810 23:06:21.922134 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:21.913344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.14.75.1:39597)
0810 23:06:21.913679 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:21.913686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:21.913711 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:06:21.913863 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:21.913867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:21.914071 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:21.914247 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:21.914703 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.915903 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.917702 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:21.918537 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:21.918642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:21.918786 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:21.921666 (+ 2880us) server_negotiation.cc:299] Negotiation successful
0810 23:06:21.921768 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":125,"threads_started":1}
I20260810 23:06:21.922299 16909 tablet_service.cc:1467] Processing CreateTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (DEFAULT_TABLE table=test-table [id=e9c159ed31594fdab433e72da2719154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:21.923487 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cfc0bde6ec4b09891c47ea4f3100a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:21.923990 16760 tablet_service.cc:1467] Processing CreateTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (DEFAULT_TABLE table=test-table [id=e9c159ed31594fdab433e72da2719154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:21.924996 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cfc0bde6ec4b09891c47ea4f3100a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:21.924815 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (DEFAULT_TABLE table=test-table [id=e9c159ed31594fdab433e72da2719154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:21.925743 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cfc0bde6ec4b09891c47ea4f3100a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:21.930682 17002 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Bootstrap starting.
I20260810 23:06:21.932806 17003 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Bootstrap starting.
I20260810 23:06:21.933466 17002 tablet_bootstrap.cc:654] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:21.934424 17002 log.cc:824] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:21.934973 17003 tablet_bootstrap.cc:654] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:21.935497 17004 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Bootstrap starting.
I20260810 23:06:21.935802 17003 log.cc:824] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:21.936380 17002 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: No bootstrap required, opened a new log
I20260810 23:06:21.936581 17002 ts_tablet_manager.cc:1397] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:06:21.937564 17004 tablet_bootstrap.cc:654] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:21.937855 17003 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: No bootstrap required, opened a new log
I20260810 23:06:21.938032 17003 ts_tablet_manager.cc:1397] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:06:21.938372 17004 log.cc:824] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:21.939929 17002 raft_consensus.cc:348] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.940171 17002 raft_consensus.cc:374] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:21.940224 17002 raft_consensus.cc:729] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Initialized, Role: FOLLOWER
I20260810 23:06:21.940387 17004 tablet_bootstrap.cc:492] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: No bootstrap required, opened a new log
I20260810 23:06:21.940562 17004 ts_tablet_manager.cc:1397] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 23:06:21.940658 17002 consensus_queue.cc:260] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.941869 17002 ts_tablet_manager.cc:1428] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:06:21.942233 16979 heartbeater.cc:503] Master 127.14.75.62:43501 was elected leader, sending a full tablet report...
I20260810 23:06:21.942224 17003 raft_consensus.cc:348] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.942476 17003 raft_consensus.cc:374] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:21.942534 17003 raft_consensus.cc:729] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Initialized, Role: FOLLOWER
I20260810 23:06:21.942979 17003 consensus_queue.cc:260] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.944135 17003 ts_tablet_manager.cc:1428] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:06:21.944489 16682 heartbeater.cc:503] Master 127.14.75.62:43501 was elected leader, sending a full tablet report...
I20260810 23:06:21.945305 17004 raft_consensus.cc:348] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.945719 17004 raft_consensus.cc:374] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:21.945804 17004 raft_consensus.cc:729] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Initialized, Role: FOLLOWER
I20260810 23:06:21.946295 17004 consensus_queue.cc:260] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:21.947233 17004 ts_tablet_manager.cc:1428] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:06:21.947638 16829 heartbeater.cc:503] Master 127.14.75.62:43501 was elected leader, sending a full tablet report...
W20260810 23:06:22.109901 16980 tablet.cc:2369] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:22.135689 16830 tablet.cc:2369] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:22.161831 16683 tablet.cc:2369] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:22.331099 17009 raft_consensus.cc:482] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:22.331310 17009 raft_consensus.cc:504] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.332661 17009 leader_election.cc:288] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:22.337833 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.333022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:35771 (local address 127.14.75.2:44961)
0810 23:06:22.333162 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.333167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.333381 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:06:22.333565 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.333568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.333618 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.333688 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:22.333968 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.334744 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.336529 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.337128 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.337239 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.337306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.337619 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.337691 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:22.337833 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.333064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:48589 (local address 127.14.75.3:45915)
0810 23:06:22.333203 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.333208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.333468 (+ 260us) server_negotiation.cc:407] Connection header received
0810 23:06:22.333804 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.333807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.333847 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.333902 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:22.334163 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.334885 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.336529 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.337128 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.337245 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.337307 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.337619 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.337691 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:22.337944 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.332953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.14.75.1:48589)
0810 23:06:22.333280 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.333293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.333453 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.333946 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.333952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.333972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:22.334065 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.334069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.334976 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.334979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.336400 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.336413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.337259 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.337262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.337455 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.337519 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.337598 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.337735 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":166,"spinlock_wait_cycles":79232,"thread_start_us":104,"threads_started":1}
I20260810 23:06:22.337955 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.332865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:44961 (local address 127.14.75.1:35771)
0810 23:06:22.333264 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.333279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.333467 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.333714 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.333716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.333732 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:22.333840 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.333845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.334856 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.334858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.336404 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.336413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.337259 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.337262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.337458 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.337517 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.337598 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.337735 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":206,"spinlock_wait_cycles":10368,"thread_start_us":132,"threads_started":1}
I20260810 23:06:22.338667 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cfc0bde6ec4b09891c47ea4f3100a7" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676" is_pre_election: true
I20260810 23:06:22.338790 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cfc0bde6ec4b09891c47ea4f3100a7" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9" is_pre_election: true
I20260810 23:06:22.339037 16781 raft_consensus.cc:2454] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 0.
I20260810 23:06:22.339138 16927 raft_consensus.cc:2454] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 0.
I20260810 23:06:22.339594 16561 leader_election.cc:302] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:22.339880 17009 raft_consensus.cc:2781] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:22.339958 17009 raft_consensus.cc:482] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:22.340006 17009 raft_consensus.cc:3037] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.341166 17009 raft_consensus.cc:504] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.341604 17009 leader_election.cc:288] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 election: Requested vote from peers a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:22.341945 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cfc0bde6ec4b09891c47ea4f3100a7" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676"
I20260810 23:06:22.341995 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cfc0bde6ec4b09891c47ea4f3100a7" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9"
I20260810 23:06:22.342051 16781 raft_consensus.cc:3037] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.342140 16927 raft_consensus.cc:3037] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.343111 16781 raft_consensus.cc:2454] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 1.
I20260810 23:06:22.343429 16561 leader_election.cc:302] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:22.343433 16927 raft_consensus.cc:2454] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 1.
I20260810 23:06:22.343591 17009 raft_consensus.cc:2781] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:22.343863 17009 raft_consensus.cc:686] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 1 LEADER]: Becoming Leader. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Running, Role: LEADER
I20260810 23:06:22.344223 17009 consensus_queue.cc:237] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.347388 16468 catalog_manager.cc:5495] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "f9eab29d1bc34310a4205519ca2dde34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:06:22 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403181, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:22.387225 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.379192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.14.75.62:43501)
0810 23:06:22.379342 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.379347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.379576 (+ 229us) server_negotiation.cc:407] Connection header received
0810 23:06:22.379689 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.379691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.379744 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.379818 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:22.380468 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.380975 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.381777 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.381979 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.382052 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.384259 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:22.384262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:22.384264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:22.384284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:22.385581 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:22.386020 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:22.386022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:22.386024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:22.386094 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:22.386325 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:22.386327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:22.386328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:22.386532 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:22.386630 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.386929 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.387091 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":52}
I20260810 23:06:22.387235 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.379151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43501 (local address 127.0.0.1:39854)
0810 23:06:22.379487 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.379500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.379610 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.379856 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.379859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.380269 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:22.380366 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.380372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.381063 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.381065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.381685 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.381694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.382072 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.382075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.382135 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.382716 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:22.382728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:22.384130 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:22.385681 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:22.385689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:22.385695 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:22.385932 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:22.386165 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:22.386167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:22.386169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:22.386248 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:22.386646 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:22.386652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:22.386759 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.386877 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.387032 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":147,"threads_started":1}
I20260810 23:06:22.391331 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:06:22.392040 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:22.396667 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 441eae1f53a149798eb90f30d19c3392 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b4133cc2f94912851fe6d4177da217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:22.396751 16760 tablet_service.cc:1467] Processing CreateTablet for tablet 441eae1f53a149798eb90f30d19c3392 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b4133cc2f94912851fe6d4177da217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:22.396848 16909 tablet_service.cc:1467] Processing CreateTablet for tablet 441eae1f53a149798eb90f30d19c3392 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c0b4133cc2f94912851fe6d4177da217]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:22.397459 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441eae1f53a149798eb90f30d19c3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.397475 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441eae1f53a149798eb90f30d19c3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.397475 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441eae1f53a149798eb90f30d19c3392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.399916 17003 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: Bootstrap starting.
I20260810 23:06:22.400472 17002 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: Bootstrap starting.
I20260810 23:06:22.400971 17003 tablet_bootstrap.cc:654] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.401046 17004 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: Bootstrap starting.
I20260810 23:06:22.401525 17002 tablet_bootstrap.cc:654] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.401856 17004 tablet_bootstrap.cc:654] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.402783 17003 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: No bootstrap required, opened a new log
I20260810 23:06:22.402809 17002 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: No bootstrap required, opened a new log
I20260810 23:06:22.402896 17003 ts_tablet_manager.cc:1397] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:06:22.402895 17002 ts_tablet_manager.cc:1397] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:06:22.403178 17004 tablet_bootstrap.cc:492] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: No bootstrap required, opened a new log
I20260810 23:06:22.403264 17004 ts_tablet_manager.cc:1397] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:22.403357 17003 raft_consensus.cc:348] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.403385 17002 raft_consensus.cc:348] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.403489 17002 raft_consensus.cc:374] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.403491 17003 raft_consensus.cc:374] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.403529 17003 raft_consensus.cc:729] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.403530 17002 raft_consensus.cc:729] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.403671 17003 consensus_queue.cc:260] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.403664 17002 consensus_queue.cc:260] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.403708 17004 raft_consensus.cc:348] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.403812 17004 raft_consensus.cc:374] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.403841 17004 raft_consensus.cc:729] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.403947 17002 ts_tablet_manager.cc:1428] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:22.403914 17009 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.403963 17004 consensus_queue.cc:260] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.404058 17009 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.404276 17004 ts_tablet_manager.cc:1428] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:22.404299 17003 ts_tablet_manager.cc:1428] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 23:06:22.404552 17008 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.404702 17008 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.405273 17010 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.405465 17010 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.580753 17008 raft_consensus.cc:482] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:22.580927 17008 raft_consensus.cc:504] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.582283 17008 leader_election.cc:288] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961)
I20260810 23:06:22.586414 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.582705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:37343 (local address 127.14.75.2:44961)
0810 23:06:22.582846 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.582851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.583115 (+ 264us) server_negotiation.cc:407] Connection header received
0810 23:06:22.583305 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.583307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.583352 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.583438 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:22.583710 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.584214 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.585286 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.585819 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.585944 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.586015 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.586202 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.586282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:22.586438 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.582584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:44961 (local address 127.14.75.3:37343)
0810 23:06:22.582981 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.582997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.583165 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.583453 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.583455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.583471 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:22.583598 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.583604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.584329 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.584334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.585149 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.585158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.585961 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.585965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.586082 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.586099 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.586185 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.586271 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260810 23:06:22.586575 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.582704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:46169 (local address 127.14.75.1:39597)
0810 23:06:22.582983 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.582988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.583282 (+ 294us) server_negotiation.cc:407] Connection header received
0810 23:06:22.583438 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.583441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.583490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.583580 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:22.583846 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.584388 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.585381 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.585958 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.586075 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.586138 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.586333 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.586410 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260810 23:06:22.586941 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441eae1f53a149798eb90f30d19c3392" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676" is_pre_election: true
I20260810 23:06:22.587064 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.582584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39597 (local address 127.14.75.3:46169)
0810 23:06:22.583162 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.583174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.583324 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.583602 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.583604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.583621 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:22.583736 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.583741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.584496 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.584501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.585258 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.585267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.586091 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.586095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.586192 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.586229 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.586314 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.586388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":253,"thread_start_us":103,"threads_started":1}
I20260810 23:06:22.587118 16781 raft_consensus.cc:2454] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 0.
I20260810 23:06:22.587502 16857 leader_election.cc:302] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9; no voters:
I20260810 23:06:22.587651 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441eae1f53a149798eb90f30d19c3392" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34" is_pre_election: true
I20260810 23:06:22.587801 17008 raft_consensus.cc:2781] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:22.587867 17008 raft_consensus.cc:482] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:22.587917 17008 raft_consensus.cc:3037] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.587929 16635 raft_consensus.cc:2454] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 0.
I20260810 23:06:22.589000 17008 raft_consensus.cc:504] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.589329 17008 leader_election.cc:288] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 election: Requested vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961)
I20260810 23:06:22.589707 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441eae1f53a149798eb90f30d19c3392" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676"
I20260810 23:06:22.589821 16781 raft_consensus.cc:3037] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.590746 16781 raft_consensus.cc:2454] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 1.
I20260810 23:06:22.591024 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441eae1f53a149798eb90f30d19c3392" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34"
I20260810 23:06:22.591085 16857 leader_election.cc:302] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9; no voters:
I20260810 23:06:22.591156 16635 raft_consensus.cc:3037] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:22.591286 17008 raft_consensus.cc:2781] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:22.591492 17008 raft_consensus.cc:686] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 LEADER]: Becoming Leader. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Running, Role: LEADER
I20260810 23:06:22.591856 17008 consensus_queue.cc:237] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.592291 16635 raft_consensus.cc:2454] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 1.
I20260810 23:06:22.594161 17027 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: TxnStatusTablet state changed. Reason: New leader c3374fef4b12417488e9cb0c4704d4f9. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.594290 17027 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:22.594588 17029 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:06:22.594806 17029 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:06:22.595361 16468 catalog_manager.cc:5495] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:22.647902 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.643758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.0.0.1:58920)
0810 23:06:22.643961 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.643976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.644076 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.644403 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.644406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.644414 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:22.644506 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.644510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.645512 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.645515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.646194 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.646210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.646770 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.646772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.646836 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.646884 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.647654 (+ 770us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.647666 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.647712 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.647786 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:06:22.647943 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.643872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.14.75.3:45915)
0810 23:06:22.644014 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.644019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.644047 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:06:22.644207 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.644210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.644286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.644387 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:22.644608 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.645391 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.646323 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.646669 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.646773 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.647019 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.647560 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.647666 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.647731 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.647789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:06:22.653153 16781 raft_consensus.cc:1261] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Refusing update from remote peer c3374fef4b12417488e9cb0c4704d4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:22.653153 16635 raft_consensus.cc:1261] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Refusing update from remote peer c3374fef4b12417488e9cb0c4704d4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:22.653914 17008 consensus_queue.cc:1035] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:22.654121 17027 consensus_queue.cc:1035] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:22.657897 17009 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: TxnStatusTablet state changed. Reason: New leader c3374fef4b12417488e9cb0c4704d4f9. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.658033 17009 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.658605 17010 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: TxnStatusTablet state changed. Reason: New leader c3374fef4b12417488e9cb0c4704d4f9. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.658754 17010 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.662034 17027 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.662159 17027 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:22.662281 17027 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.662346 17027 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P c3374fef4b12417488e9cb0c4704d4f9: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:22.662801 17009 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.662914 17009 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P f9eab29d1bc34310a4205519ca2dde34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.664139 17010 tablet_replica.cc:439] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } } }
I20260810 23:06:22.664289 17010 tablet_replica.cc:442] T 441eae1f53a149798eb90f30d19c3392 P a01e2d092baf4d809c66ee08b242a676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:22.665321 17034 mvcc.cc:205] Tried to move back new op lower bound from 7317107436140240896 to 7317107435899326464. Current Snapshot: MvccSnapshot[applied={T|T < 7317107436140240896}]
I20260810 23:06:22.673619 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.669571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.14.75.1:39597)
0810 23:06:22.669685 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.669690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.670084 (+ 394us) server_negotiation.cc:407] Connection header received
0810 23:06:22.670280 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.670283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.670351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.670465 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:22.670725 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.671238 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.672124 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.672365 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.672476 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.672757 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.673190 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.673316 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.673384 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.673467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:06:22.685032 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.681112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:36393 (local address 127.14.75.3:45915)
0810 23:06:22.681237 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.681241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.681545 (+ 304us) server_negotiation.cc:407] Connection header received
0810 23:06:22.681647 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.681650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.681710 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.681793 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:22.682044 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.682827 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.683838 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.684185 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.684299 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.684597 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.684683 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.684755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.684844 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.684897 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:06:22.685052 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.681002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.14.75.1:36393)
0810 23:06:22.681397 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.681454 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.681562 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.681809 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.681812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.681818 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:22.681936 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.681941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.682966 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.682969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.683708 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.683718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.684318 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.684321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.684393 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.684442 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.684767 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.684772 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.684828 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.684893 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":117,"threads_started":1}
I20260810 23:06:22.693092 16781 raft_consensus.cc:1261] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Refusing update from remote peer f9eab29d1bc34310a4205519ca2dde34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:22.693219 16927 raft_consensus.cc:1261] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Refusing update from remote peer f9eab29d1bc34310a4205519ca2dde34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:22.693756 17013 consensus_queue.cc:1035] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:22.693969 17009 consensus_queue.cc:1035] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:22.698269 17032 mvcc.cc:205] Tried to move back new op lower bound from 7317107436305981440 to 7317107434884530176. Current Snapshot: MvccSnapshot[applied={T|T < 7317107436305981440}]
I20260810 23:06:22.703212 17033 mvcc.cc:205] Tried to move back new op lower bound from 7317107436305981440 to 7317107434884530176. Current Snapshot: MvccSnapshot[applied={T|T < 7317107436305981440}]
I20260810 23:06:22.718761 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.715150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:51929 (local address 127.14.75.1:39597)
0810 23:06:22.715294 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:22.715300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:22.715580 (+ 280us) server_negotiation.cc:407] Connection header received
0810 23:06:22.715731 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:22.715734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:22.715782 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:22.715861 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:22.716185 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.716707 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.717671 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.717889 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.718000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.718313 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.718389 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.718463 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:22.718561 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:06:22.718612 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:22.718827 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:22.715051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39597 (local address 127.14.75.3:51929)
0810 23:06:22.715500 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:22.715517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:22.715626 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:22.715871 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:22.715875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:22.715883 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:22.716041 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.716049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.716843 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.716847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.717546 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:22.717555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:22.718025 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:22.718028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:22.718135 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:22.718179 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:22.718483 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:22.718489 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:06:22.718550 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:06:22.718637 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":131,"threads_started":1}
I20260810 23:06:22.749691 16468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:22.749908 16468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:22.752768 16468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Sending DeleteTablet for 3 replicas of tablet 83cfc0bde6ec4b09891c47ea4f3100a7
I20260810 23:06:22.753664 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:22 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:42592
I20260810 23:06:22.754252 17054 tablet_replica.cc:330] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: stopping tablet replica
I20260810 23:06:22.754333 16909 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:22 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:58912
I20260810 23:06:22.754753 17054 raft_consensus.cc:2229] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:22.755213 17054 raft_consensus.cc:2258] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:22.755661 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:22.756119 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:22.756943 17054 ts_tablet_manager.cc:1905] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:22.760293 17054 ts_tablet_manager.cc:1918] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:22.760488 17054 log.cc:1196] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/83cfc0bde6ec4b09891c47ea4f3100a7
I20260810 23:06:22.760892 17054 ts_tablet_manager.cc:1939] T 83cfc0bde6ec4b09891c47ea4f3100a7 P f9eab29d1bc34310a4205519ca2dde34: Deleting consensus metadata
I20260810 23:06:22.761742 16615 tablet_service.cc:1467] Processing CreateTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (DEFAULT_TABLE table=test-table [id=1194a7ee072042b89d260ef0249b4f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:22.761739 16909 tablet_service.cc:1467] Processing CreateTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (DEFAULT_TABLE table=test-table [id=1194a7ee072042b89d260ef0249b4f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:22.762001 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.762013 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.764640 17002 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Bootstrap starting.
I20260810 23:06:22.765471 17003 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Bootstrap starting.
I20260810 23:06:22.765730 17002 tablet_bootstrap.cc:654] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.766433 17003 tablet_bootstrap.cc:654] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.767740 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cfc0bde6ec4b09891c47ea4f3100a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:22 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45858
I20260810 23:06:22.767920 16453 catalog_manager.cc:4841] TS f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597): tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (table test-table [id=e9c159ed31594fdab433e72da2719154]) successfully deleted
I20260810 23:06:22.768469 16760 tablet_service.cc:1467] Processing CreateTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (DEFAULT_TABLE table=test-table [id=1194a7ee072042b89d260ef0249b4f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:22.768786 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:22.770165 17058 tablet_replica.cc:330] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: stopping tablet replica
I20260810 23:06:22.770557 17058 raft_consensus.cc:2229] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:22.770833 17058 raft_consensus.cc:2258] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:22.771675 17003 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: No bootstrap required, opened a new log
I20260810 23:06:22.771780 17003 ts_tablet_manager.cc:1397] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 23:06:22.771891 17002 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: No bootstrap required, opened a new log
I20260810 23:06:22.771977 17002 ts_tablet_manager.cc:1397] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260810 23:06:22.772223 17003 raft_consensus.cc:348] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.772332 17003 raft_consensus.cc:374] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.772359 17003 raft_consensus.cc:729] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.772413 17002 raft_consensus.cc:348] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.772471 17003 consensus_queue.cc:260] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.772521 17002 raft_consensus.cc:374] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.772553 17002 raft_consensus.cc:729] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.772704 17003 ts_tablet_manager.cc:1428] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:22.772693 17002 consensus_queue.cc:260] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.772964 17002 ts_tablet_manager.cc:1428] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:22.774516 17004 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Bootstrap starting.
I20260810 23:06:22.774583 17055 tablet_replica.cc:330] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: stopping tablet replica
I20260810 23:06:22.774904 17055 raft_consensus.cc:2229] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:22.775100 17055 raft_consensus.cc:2258] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:22.775084 17058 ts_tablet_manager.cc:1905] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:22.775616 17004 tablet_bootstrap.cc:654] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:22.776675 17055 ts_tablet_manager.cc:1905] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:22.777143 17004 tablet_bootstrap.cc:492] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: No bootstrap required, opened a new log
I20260810 23:06:22.777254 17004 ts_tablet_manager.cc:1397] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:22.777758 17004 raft_consensus.cc:348] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.777879 17004 raft_consensus.cc:374] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:22.777920 17004 raft_consensus.cc:729] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Initialized, Role: FOLLOWER
I20260810 23:06:22.778054 17004 consensus_queue.cc:260] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:22.778350 17004 ts_tablet_manager.cc:1428] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:22.780644 17058 ts_tablet_manager.cc:1918] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:22.780746 17058 log.cc:1196] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/83cfc0bde6ec4b09891c47ea4f3100a7
I20260810 23:06:22.780928 17055 ts_tablet_manager.cc:1918] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:22.781020 17055 log.cc:1196] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/83cfc0bde6ec4b09891c47ea4f3100a7
I20260810 23:06:22.781100 17058 ts_tablet_manager.cc:1939] T 83cfc0bde6ec4b09891c47ea4f3100a7 P a01e2d092baf4d809c66ee08b242a676: Deleting consensus metadata
I20260810 23:06:22.781353 17055 ts_tablet_manager.cc:1939] T 83cfc0bde6ec4b09891c47ea4f3100a7 P c3374fef4b12417488e9cb0c4704d4f9: Deleting consensus metadata
I20260810 23:06:22.782476 16455 catalog_manager.cc:4841] TS c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915): tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (table test-table [id=e9c159ed31594fdab433e72da2719154]) successfully deleted
I20260810 23:06:22.782476 16451 catalog_manager.cc:4841] TS a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961): tablet 83cfc0bde6ec4b09891c47ea4f3100a7 (table test-table [id=e9c159ed31594fdab433e72da2719154]) successfully deleted
I20260810 23:06:23.028513 17008 raft_consensus.cc:482] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:23.028693 17008 raft_consensus.cc:504] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:23.029186 17008 leader_election.cc:288] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961)
I20260810 23:06:23.029749 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a9f771ddac4d7c9fe2f3c54d5e0020" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34" is_pre_election: true
I20260810 23:06:23.029752 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a9f771ddac4d7c9fe2f3c54d5e0020" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676" is_pre_election: true
I20260810 23:06:23.029925 16635 raft_consensus.cc:2454] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 0.
I20260810 23:06:23.029932 16781 raft_consensus.cc:2454] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 0.
I20260810 23:06:23.030272 16859 leader_election.cc:302] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3374fef4b12417488e9cb0c4704d4f9, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:23.030498 17008 raft_consensus.cc:2781] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:23.030567 17008 raft_consensus.cc:482] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:23.030630 17008 raft_consensus.cc:3037] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:23.031724 17008 raft_consensus.cc:504] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:23.032107 17008 leader_election.cc:288] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 election: Requested vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961)
I20260810 23:06:23.032496 16635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a9f771ddac4d7c9fe2f3c54d5e0020" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34"
I20260810 23:06:23.032506 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a9f771ddac4d7c9fe2f3c54d5e0020" candidate_uuid: "c3374fef4b12417488e9cb0c4704d4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676"
I20260810 23:06:23.032626 16635 raft_consensus.cc:3037] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:23.032640 16781 raft_consensus.cc:3037] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:23.033668 16635 raft_consensus.cc:2454] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 1.
I20260810 23:06:23.033666 16781 raft_consensus.cc:2454] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3374fef4b12417488e9cb0c4704d4f9 in term 1.
I20260810 23:06:23.033994 16859 leader_election.cc:302] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3374fef4b12417488e9cb0c4704d4f9, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:23.034231 17008 raft_consensus.cc:2781] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:23.034320 17008 raft_consensus.cc:686] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 LEADER]: Becoming Leader. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Running, Role: LEADER
I20260810 23:06:23.034467 17008 consensus_queue.cc:237] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:23.036096 16468 catalog_manager.cc:5495] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "c3374fef4b12417488e9cb0c4704d4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:23.079133 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:23.076538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.14.75.3:45915)
0810 23:06:23.076662 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:23.076667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:23.076679 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:23.076772 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:23.076774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:23.076814 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:23.076874 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:23.077027 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.077541 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.078299 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.078499 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.078571 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:23.078777 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:23.078842 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:23.078894 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:23.078976 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 23:06:23.079015 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:06:23.087333 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:23.084514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.14.75.3:53645)
0810 23:06:23.084655 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:23.084668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:23.084742 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:23.084937 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:23.084939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:23.084944 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:23.085018 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:23.085022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.085661 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.085663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:23.086217 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:23.086225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.086603 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.086605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:23.086696 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:23.086753 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:23.087031 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:23.087036 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:23.087089 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:06:23.087154 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:06:23.087339 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:23.084588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53645 (local address 127.14.75.3:45915)
0810 23:06:23.084696 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:23.084699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:23.084711 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:23.084822 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:23.084824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:23.084861 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:23.084921 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:23.085098 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.085583 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.086317 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:23.086513 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:23.086589 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:23.086872 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:23.086955 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:23.087012 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:23.087135 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 23:06:23.087179 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:23.092983 16635 raft_consensus.cc:1261] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Refusing update from remote peer c3374fef4b12417488e9cb0c4704d4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:23.092983 16781 raft_consensus.cc:1261] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Refusing update from remote peer c3374fef4b12417488e9cb0c4704d4f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:23.093430 17008 consensus_queue.cc:1035] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:23.093616 17027 consensus_queue.cc:1035] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:06:23.111574 17032 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:06:24.667256 16468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:24.667413 16468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:24.669966 16468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Sending DeleteTablet for 3 replicas of tablet a2a9f771ddac4d7c9fe2f3c54d5e0020
I20260810 23:06:24.670598 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45858
I20260810 23:06:24.670642 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:42592
I20260810 23:06:24.671015 17088 tablet_replica.cc:330] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: stopping tablet replica
I20260810 23:06:24.671088 16909 tablet_service.cc:1514] Processing DeleteTablet for tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:58912
I20260810 23:06:24.671211 17088 raft_consensus.cc:2229] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:24.671361 17088 raft_consensus.cc:2258] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.671445 17089 tablet_replica.cc:330] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: stopping tablet replica
I20260810 23:06:24.671670 17089 raft_consensus.cc:2229] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:24.671998 17088 ts_tablet_manager.cc:1905] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.672066 17089 raft_consensus.cc:2258] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.672818 17089 ts_tablet_manager.cc:1905] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.674953 17088 ts_tablet_manager.cc:1918] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:06:24.675068 17088 log.cc:1196] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/a2a9f771ddac4d7c9fe2f3c54d5e0020
I20260810 23:06:24.675355 17088 ts_tablet_manager.cc:1939] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P f9eab29d1bc34310a4205519ca2dde34: Deleting consensus metadata
I20260810 23:06:24.676039 16453 catalog_manager.cc:4841] TS f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597): tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (table test-table [id=1194a7ee072042b89d260ef0249b4f77]) successfully deleted
I20260810 23:06:24.676216 17089 ts_tablet_manager.cc:1918] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:06:24.676317 17089 log.cc:1196] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/a2a9f771ddac4d7c9fe2f3c54d5e0020
I20260810 23:06:24.676658 17089 ts_tablet_manager.cc:1939] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P c3374fef4b12417488e9cb0c4704d4f9: Deleting consensus metadata
I20260810 23:06:24.677500 16455 catalog_manager.cc:4841] TS c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915): tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (table test-table [id=1194a7ee072042b89d260ef0249b4f77]) successfully deleted
I20260810 23:06:24.677634 17087 tablet_replica.cc:330] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: stopping tablet replica
I20260810 23:06:24.677778 17087 raft_consensus.cc:2229] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:24.677914 17087 raft_consensus.cc:2258] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.678505 17087 ts_tablet_manager.cc:1905] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.681131 17087 ts_tablet_manager.cc:1918] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:06:24.681226 17087 log.cc:1196] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/a2a9f771ddac4d7c9fe2f3c54d5e0020
I20260810 23:06:24.681524 17087 ts_tablet_manager.cc:1939] T a2a9f771ddac4d7c9fe2f3c54d5e0020 P a01e2d092baf4d809c66ee08b242a676: Deleting consensus metadata
I20260810 23:06:24.682240 16451 catalog_manager.cc:4841] TS a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961): tablet a2a9f771ddac4d7c9fe2f3c54d5e0020 (table test-table [id=1194a7ee072042b89d260ef0249b4f77]) successfully deleted
Aug 10 23:06:24 dist-test-slave-srgv krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:06:24 dist-test-slave-srgv krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:24.706432 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.691491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.14.75.62:43501)
0810 23:06:24.691817 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.691827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.691843 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:24.691993 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.691996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.692052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.692147 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:24.692891 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.693519 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.694405 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.694654 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.694757 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.703316 (+ 8559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:24.703320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:24.703322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:24.703352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:24.704776 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:24.705166 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:24.705169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:24.705170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:24.705248 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:24.705551 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:24.705554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:24.705555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:24.705774 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:24.705886 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.706101 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.706213 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
I20260810 23:06:24.709107 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:24.709636 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:24.714229 16615 tablet_service.cc:1467] Processing CreateTablet for tablet a5e44cc814354f249a4ead59f21b5a84 (DEFAULT_TABLE table=test-table [id=923986c3c52c4153ad2d44289d90aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.714274 16760 tablet_service.cc:1467] Processing CreateTablet for tablet a5e44cc814354f249a4ead59f21b5a84 (DEFAULT_TABLE table=test-table [id=923986c3c52c4153ad2d44289d90aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.714442 16909 tablet_service.cc:1467] Processing CreateTablet for tablet a5e44cc814354f249a4ead59f21b5a84 (DEFAULT_TABLE table=test-table [id=923986c3c52c4153ad2d44289d90aaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.714561 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e44cc814354f249a4ead59f21b5a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.714561 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e44cc814354f249a4ead59f21b5a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.714735 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e44cc814354f249a4ead59f21b5a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.716708 17099 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Bootstrap starting.
I20260810 23:06:24.717178 17102 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Bootstrap starting.
I20260810 23:06:24.717370 17101 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Bootstrap starting.
I20260810 23:06:24.717669 17099 tablet_bootstrap.cc:654] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.718133 17102 tablet_bootstrap.cc:654] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.718369 17101 tablet_bootstrap.cc:654] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.718878 17099 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: No bootstrap required, opened a new log
I20260810 23:06:24.718983 17099 ts_tablet_manager.cc:1397] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:24.719385 17102 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: No bootstrap required, opened a new log
I20260810 23:06:24.719489 17102 ts_tablet_manager.cc:1397] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:24.719436 17099 raft_consensus.cc:348] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.719566 17099 raft_consensus.cc:374] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.719614 17099 raft_consensus.cc:729] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.719754 17099 consensus_queue.cc:260] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.719940 17102 raft_consensus.cc:348] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.720049 17102 raft_consensus.cc:374] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.720046 17099 ts_tablet_manager.cc:1428] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:24.720084 17102 raft_consensus.cc:729] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.720182 17101 tablet_bootstrap.cc:492] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: No bootstrap required, opened a new log
I20260810 23:06:24.720261 17101 ts_tablet_manager.cc:1397] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:06:24.720235 17102 consensus_queue.cc:260] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.720605 17102 ts_tablet_manager.cc:1428] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:24.720731 17101 raft_consensus.cc:348] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.720846 17101 raft_consensus.cc:374] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.720913 17101 raft_consensus.cc:729] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.721072 17101 consensus_queue.cc:260] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.721527 17101 ts_tablet_manager.cc:1428] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:24.767423 17107 raft_consensus.cc:482] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:24.767580 17107 raft_consensus.cc:504] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.768923 17107 leader_election.cc:288] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:24.773284 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.769247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:38957 (local address 127.14.75.1:39597)
0810 23:06:24.769605 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.769612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.769737 (+ 125us) server_negotiation.cc:407] Connection header received
0810 23:06:24.769842 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.769845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.769895 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.769987 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:24.770332 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.770888 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772209 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.772617 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772738 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.772809 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.773051 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.773132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":146,"threads_started":1}
I20260810 23:06:24.773298 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.769247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.14.75.2:43467)
0810 23:06:24.769762 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:24.769775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:24.769884 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:24.770181 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:24.770183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:24.770197 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:24.770328 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.770334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.771183 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.771185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.772037 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.772045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.772772 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.772927 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.772944 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:24.773033 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:06:24.773118 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":42,"thread_start_us":115,"threads_started":1}
I20260810 23:06:24.773284 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.769348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:43467 (local address 127.14.75.3:45915)
0810 23:06:24.769686 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.769696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.769871 (+ 175us) server_negotiation.cc:407] Connection header received
0810 23:06:24.770019 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.770022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.770072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.770166 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:24.770439 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.771054 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772165 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.772613 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772738 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.772808 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.773056 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.773145 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":134,"threads_started":1}
I20260810 23:06:24.773299 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.769140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39597 (local address 127.14.75.2:38957)
0810 23:06:24.769595 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:24.769611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:24.769737 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:24.770015 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:24.770019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:24.770039 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:24.770203 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.770210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.771082 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.771088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.772058 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.772068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.772768 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.772772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.772915 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.772940 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:06:24.773034 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:06:24.773141 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":116,"thread_start_us":146,"threads_started":1}
I20260810 23:06:24.774044 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e44cc814354f249a4ead59f21b5a84" candidate_uuid: "a01e2d092baf4d809c66ee08b242a676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34" is_pre_election: true
I20260810 23:06:24.774252 16634 raft_consensus.cc:2454] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a01e2d092baf4d809c66ee08b242a676 in term 0.
I20260810 23:06:24.774225 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e44cc814354f249a4ead59f21b5a84" candidate_uuid: "a01e2d092baf4d809c66ee08b242a676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9" is_pre_election: true
I20260810 23:06:24.774394 16927 raft_consensus.cc:2454] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a01e2d092baf4d809c66ee08b242a676 in term 0.
I20260810 23:06:24.774801 16711 leader_election.cc:302] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:24.775115 17107 raft_consensus.cc:2781] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:24.775197 17107 raft_consensus.cc:482] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:24.775252 17107 raft_consensus.cc:3037] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:24.776324 17107 raft_consensus.cc:504] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.776698 17107 leader_election.cc:288] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [CANDIDATE]: Term 1 election: Requested vote from peers f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:24.777062 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e44cc814354f249a4ead59f21b5a84" candidate_uuid: "a01e2d092baf4d809c66ee08b242a676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9"
I20260810 23:06:24.777199 16927 raft_consensus.cc:3037] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:24.777168 16634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e44cc814354f249a4ead59f21b5a84" candidate_uuid: "a01e2d092baf4d809c66ee08b242a676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9eab29d1bc34310a4205519ca2dde34"
I20260810 23:06:24.777294 16634 raft_consensus.cc:3037] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:24.778445 16634 raft_consensus.cc:2454] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a01e2d092baf4d809c66ee08b242a676 in term 1.
I20260810 23:06:24.778445 16927 raft_consensus.cc:2454] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a01e2d092baf4d809c66ee08b242a676 in term 1.
I20260810 23:06:24.778754 16713 leader_election.cc:302] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9; no voters:
I20260810 23:06:24.778950 17107 raft_consensus.cc:2781] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:24.779203 17107 raft_consensus.cc:686] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 1 LEADER]: Becoming Leader. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Running, Role: LEADER
I20260810 23:06:24.779450 17107 consensus_queue.cc:237] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.781883 16467 catalog_manager.cc:5495] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 reported cstate change: term changed from 0 to 1, leader changed from <none> to a01e2d092baf4d809c66ee08b242a676 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "a01e2d092baf4d809c66ee08b242a676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:24.809162 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.805294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.14.75.2:44961)
0810 23:06:24.805595 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.805602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.805614 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:24.805685 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.805687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.805742 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.805846 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:24.806037 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.806639 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.807669 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.807921 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.808007 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.808254 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:24.808724 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:24.808816 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.808898 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.808946 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":151,"threads_started":1}
I20260810 23:06:24.819075 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.816239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:53909 (local address 127.14.75.3:45915)
0810 23:06:24.816370 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.816373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.816387 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:24.816465 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.816467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.816513 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.816570 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:24.816810 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.817300 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.818176 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.818391 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.818469 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.818725 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:24.818789 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:24.818843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.818923 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.818967 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:06:24.819100 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.815967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45915 (local address 127.14.75.2:53909)
0810 23:06:24.816284 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:24.816297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:24.816377 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:24.816593 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:24.816595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:24.816601 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:24.816714 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.816717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.817388 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.817390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.818064 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.818073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.818491 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.818494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.818567 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.818610 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:24.818864 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:24.818868 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:06:24.818915 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:06:24.818979 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260810 23:06:24.825470 16634 raft_consensus.cc:1261] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Refusing update from remote peer a01e2d092baf4d809c66ee08b242a676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:24.825470 16927 raft_consensus.cc:1261] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Refusing update from remote peer a01e2d092baf4d809c66ee08b242a676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:24.825942 17112 consensus_queue.cc:1035] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:24.826143 17107 consensus_queue.cc:1035] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:24.843832 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.840298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:58757 (local address 127.14.75.2:44961)
0810 23:06:24.840420 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:24.840424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:24.840603 (+ 179us) server_negotiation.cc:407] Connection header received
0810 23:06:24.840713 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:24.840714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:24.840756 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:24.840813 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:24.841068 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.841575 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.842745 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.843074 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.843185 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.843467 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:24.843537 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:24.843595 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:24.843680 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 23:06:24.843726 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:06:24.843848 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:24.840199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:44961 (local address 127.14.75.3:58757)
0810 23:06:24.840539 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:24.840552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:24.840637 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:24.840829 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:24.840831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:24.840837 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:24.841000 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.841006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.841689 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.841692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.842612 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:24.842622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:24.843197 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:24.843203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:24.843305 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:24.843349 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:24.843614 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:24.843619 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:24.843666 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:06:24.843736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":115,"threads_started":1}
I20260810 23:06:24.867064 16468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:24.867252 16468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:24.869626 16468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Sending DeleteTablet for 3 replicas of tablet a5e44cc814354f249a4ead59f21b5a84
I20260810 23:06:24.870186 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e44cc814354f249a4ead59f21b5a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:42592
I20260810 23:06:24.870204 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e44cc814354f249a4ead59f21b5a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45858
I20260810 23:06:24.870368 17088 tablet_replica.cc:330] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: stopping tablet replica
I20260810 23:06:24.870378 17087 tablet_replica.cc:330] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: stopping tablet replica
I20260810 23:06:24.870481 17088 raft_consensus.cc:2229] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:24.870482 17087 raft_consensus.cc:2229] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:24.870601 17088 raft_consensus.cc:2258] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.870783 17087 raft_consensus.cc:2258] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.871609 17088 ts_tablet_manager.cc:1905] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.874392 17088 ts_tablet_manager.cc:1918] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:24.874478 17088 log.cc:1196] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/a5e44cc814354f249a4ead59f21b5a84
I20260810 23:06:24.875080 17087 ts_tablet_manager.cc:1905] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.877575 17087 ts_tablet_manager.cc:1918] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:24.877660 17087 log.cc:1196] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/a5e44cc814354f249a4ead59f21b5a84
I20260810 23:06:24.877805 17088 ts_tablet_manager.cc:1939] T a5e44cc814354f249a4ead59f21b5a84 P f9eab29d1bc34310a4205519ca2dde34: Deleting consensus metadata
I20260810 23:06:24.877921 17087 ts_tablet_manager.cc:1939] T a5e44cc814354f249a4ead59f21b5a84 P a01e2d092baf4d809c66ee08b242a676: Deleting consensus metadata
I20260810 23:06:24.878645 16453 catalog_manager.cc:4841] TS f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597): tablet a5e44cc814354f249a4ead59f21b5a84 (table test-table [id=923986c3c52c4153ad2d44289d90aaf4]) successfully deleted
I20260810 23:06:24.878717 16451 catalog_manager.cc:4841] TS a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961): tablet a5e44cc814354f249a4ead59f21b5a84 (table test-table [id=923986c3c52c4153ad2d44289d90aaf4]) successfully deleted
I20260810 23:06:24.879045 16909 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e44cc814354f249a4ead59f21b5a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:24 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:58912
I20260810 23:06:24.879221 17089 tablet_replica.cc:330] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: stopping tablet replica
I20260810 23:06:24.879324 17089 raft_consensus.cc:2229] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:24.879438 17089 raft_consensus.cc:2258] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:24.880162 17089 ts_tablet_manager.cc:1905] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:24.880123 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:24.880503 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:24.882714 17089 ts_tablet_manager.cc:1918] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:24.882791 17089 log.cc:1196] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/a5e44cc814354f249a4ead59f21b5a84
I20260810 23:06:24.883037 17089 ts_tablet_manager.cc:1939] T a5e44cc814354f249a4ead59f21b5a84 P c3374fef4b12417488e9cb0c4704d4f9: Deleting consensus metadata
I20260810 23:06:24.884132 16455 catalog_manager.cc:4841] TS c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915): tablet a5e44cc814354f249a4ead59f21b5a84 (table test-table [id=923986c3c52c4153ad2d44289d90aaf4]) successfully deleted
I20260810 23:06:24.885175 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 740e9466b9f84414bc8f51230f2c885f (DEFAULT_TABLE table=test-table [id=9cf2eab0bc374acca5c263b4de950638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.885330 16760 tablet_service.cc:1467] Processing CreateTablet for tablet 740e9466b9f84414bc8f51230f2c885f (DEFAULT_TABLE table=test-table [id=9cf2eab0bc374acca5c263b4de950638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.885542 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e9466b9f84414bc8f51230f2c885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.885640 16760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e9466b9f84414bc8f51230f2c885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.887413 17102 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34: Bootstrap starting.
I20260810 23:06:24.887930 17101 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Bootstrap starting.
I20260810 23:06:24.888191 17102 tablet_bootstrap.cc:654] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.888840 17101 tablet_bootstrap.cc:654] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.889367 17102 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34: No bootstrap required, opened a new log
I20260810 23:06:24.889493 17102 ts_tablet_manager.cc:1397] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 23:06:24.889860 17102 raft_consensus.cc:348] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.889988 17102 raft_consensus.cc:374] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.890030 17102 raft_consensus.cc:729] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.890169 17102 consensus_queue.cc:260] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.890246 16909 tablet_service.cc:1467] Processing CreateTablet for tablet 740e9466b9f84414bc8f51230f2c885f (DEFAULT_TABLE table=test-table [id=9cf2eab0bc374acca5c263b4de950638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:24.890455 17102 ts_tablet_manager.cc:1428] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:24.890512 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740e9466b9f84414bc8f51230f2c885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:24.892844 17099 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Bootstrap starting.
I20260810 23:06:24.892968 17101 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: No bootstrap required, opened a new log
I20260810 23:06:24.893046 17101 ts_tablet_manager.cc:1397] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:06:24.893738 17099 tablet_bootstrap.cc:654] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:24.894294 17101 raft_consensus.cc:348] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.894424 17101 raft_consensus.cc:374] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.894469 17101 raft_consensus.cc:729] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01e2d092baf4d809c66ee08b242a676, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.894601 17101 consensus_queue.cc:260] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.894882 17101 ts_tablet_manager.cc:1428] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:24.895726 17099 tablet_bootstrap.cc:492] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: No bootstrap required, opened a new log
I20260810 23:06:24.895823 17099 ts_tablet_manager.cc:1397] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:24.896265 17099 raft_consensus.cc:348] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.896368 17099 raft_consensus.cc:374] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:24.896390 17099 raft_consensus.cc:729] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3374fef4b12417488e9cb0c4704d4f9, State: Initialized, Role: FOLLOWER
I20260810 23:06:24.896521 17099 consensus_queue.cc:260] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:24.896808 17099 ts_tablet_manager.cc:1428] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:25.039767 17106 raft_consensus.cc:482] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:25.040009 17106 raft_consensus.cc:504] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:25.040522 17106 leader_election.cc:288] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:25.041004 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e9466b9f84414bc8f51230f2c885f" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676" is_pre_election: true
I20260810 23:06:25.041059 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e9466b9f84414bc8f51230f2c885f" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9" is_pre_election: true
I20260810 23:06:25.041181 16781 raft_consensus.cc:2454] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 0.
I20260810 23:06:25.041206 16927 raft_consensus.cc:2454] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 0.
I20260810 23:06:25.041580 16564 leader_election.cc:302] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, c3374fef4b12417488e9cb0c4704d4f9, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:25.041779 17106 raft_consensus.cc:2781] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:25.041846 17106 raft_consensus.cc:482] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:25.041873 17106 raft_consensus.cc:3037] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:25.042948 17106 raft_consensus.cc:504] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:25.043308 17106 leader_election.cc:288] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 election: Requested vote from peers a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961), c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915)
I20260810 23:06:25.043681 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e9466b9f84414bc8f51230f2c885f" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a01e2d092baf4d809c66ee08b242a676"
I20260810 23:06:25.043680 16927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740e9466b9f84414bc8f51230f2c885f" candidate_uuid: "f9eab29d1bc34310a4205519ca2dde34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3374fef4b12417488e9cb0c4704d4f9"
I20260810 23:06:25.043794 16781 raft_consensus.cc:3037] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:25.043826 16927 raft_consensus.cc:3037] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:25.044737 16781 raft_consensus.cc:2454] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 1.
I20260810 23:06:25.044757 16927 raft_consensus.cc:2454] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9eab29d1bc34310a4205519ca2dde34 in term 1.
I20260810 23:06:25.045040 16561 leader_election.cc:302] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a01e2d092baf4d809c66ee08b242a676, f9eab29d1bc34310a4205519ca2dde34; no voters:
I20260810 23:06:25.045252 17106 raft_consensus.cc:2781] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:25.045486 17106 raft_consensus.cc:686] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [term 1 LEADER]: Becoming Leader. State: Replica: f9eab29d1bc34310a4205519ca2dde34, State: Running, Role: LEADER
I20260810 23:06:25.045644 17106 consensus_queue.cc:237] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } }
I20260810 23:06:25.047264 16468 catalog_manager.cc:5495] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "f9eab29d1bc34310a4205519ca2dde34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9eab29d1bc34310a4205519ca2dde34" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:25.095968 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:25.092758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.14.75.1:39597)
0810 23:06:25.092880 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:25.092886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:25.092923 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:25.093091 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:25.093094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:25.093138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:25.093235 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:25.093515 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:25.094025 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:25.094856 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:25.095085 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:25.095212 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:25.095464 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:25.095576 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:25.095655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:25.095760 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 23:06:25.095814 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:06:25.106645 16781 raft_consensus.cc:1261] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Refusing update from remote peer f9eab29d1bc34310a4205519ca2dde34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:25.106648 16927 raft_consensus.cc:1261] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Refusing update from remote peer f9eab29d1bc34310a4205519ca2dde34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:25.107079 17129 consensus_queue.cc:1035] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3374fef4b12417488e9cb0c4704d4f9" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:25.107522 17129 consensus_queue.cc:1035] T 740e9466b9f84414bc8f51230f2c885f P f9eab29d1bc34310a4205519ca2dde34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a01e2d092baf4d809c66ee08b242a676" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:06:25.121614 17071 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:06:26.674628 16467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:26.674811 16467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:26.677322 16467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a52e3cfdd9422aade931788bdf87b9: Sending DeleteTablet for 3 replicas of tablet 740e9466b9f84414bc8f51230f2c885f
I20260810 23:06:26.677959 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e9466b9f84414bc8f51230f2c885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:26 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:42592
I20260810 23:06:26.678006 16760 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e9466b9f84414bc8f51230f2c885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:26 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45858
I20260810 23:06:26.678300 16909 tablet_service.cc:1514] Processing DeleteTablet for tablet 740e9466b9f84414bc8f51230f2c885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:26 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:58912
I20260810 23:06:26.678390 17159 tablet_replica.cc:330] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: stopping tablet replica
I20260810 23:06:26.678504 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16544
I20260810 23:06:26.680644 17159 raft_consensus.cc:2229] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:26.680817 17159 raft_consensus.cc:2258] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:26.681113 17160 tablet_replica.cc:330] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: stopping tablet replica
I20260810 23:06:26.681295 17160 raft_consensus.cc:2229] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:26.681567 17160 raft_consensus.cc:2258] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:26.681574 17159 ts_tablet_manager.cc:1905] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:26.682303 17160 ts_tablet_manager.cc:1905] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:26.684504 17159 ts_tablet_manager.cc:1918] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:06:26.684599 17159 log.cc:1196] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/740e9466b9f84414bc8f51230f2c885f
I20260810 23:06:26.684942 17159 ts_tablet_manager.cc:1939] T 740e9466b9f84414bc8f51230f2c885f P a01e2d092baf4d809c66ee08b242a676: Deleting consensus metadata
I20260810 23:06:26.685158 17160 ts_tablet_manager.cc:1918] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:06:26.685240 17160 log.cc:1196] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/740e9466b9f84414bc8f51230f2c885f
I20260810 23:06:26.685573 17160 ts_tablet_manager.cc:1939] T 740e9466b9f84414bc8f51230f2c885f P c3374fef4b12417488e9cb0c4704d4f9: Deleting consensus metadata
I20260810 23:06:26.685801 16451 catalog_manager.cc:4841] TS a01e2d092baf4d809c66ee08b242a676 (127.14.75.2:44961): tablet 740e9466b9f84414bc8f51230f2c885f (table test-table [id=9cf2eab0bc374acca5c263b4de950638]) successfully deleted
I20260810 23:06:26.686276 16455 catalog_manager.cc:4841] TS c3374fef4b12417488e9cb0c4704d4f9 (127.14.75.3:45915): tablet 740e9466b9f84414bc8f51230f2c885f (table test-table [id=9cf2eab0bc374acca5c263b4de950638]) successfully deleted
W20260810 23:06:26.687513 16453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:39597 (error 108)
I20260810 23:06:26.688424 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16692
I20260810 23:06:26.689687 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:26.688236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39597 (local address 127.0.0.1:38692)
0810 23:06:26.689468 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:26.689563 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
W20260810 23:06:26.689918 16453 catalog_manager.cc:4584] TS f9eab29d1bc34310a4205519ca2dde34 (127.14.75.1:39597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 740e9466b9f84414bc8f51230f2c885f: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39597: connect: Connection refused (error 111)
I20260810 23:06:26.700275 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16841
I20260810 23:06:26.708982 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 16435
2026-08-10T23:06:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7004 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:26 dist-test-slave-srgv krb5kdc[17165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:26 dist-test-slave-srgv krb5kdc[17165](info): set up 2 sockets
Aug 10 23:06:26 dist-test-slave-srgv krb5kdc[17165](info): commencing operation
krb5kdc: starting...
W20260810 23:06:27.774552 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:27 dist-test-slave-srgv krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:27.903982 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42399
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:28.021521 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:28.021811 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:28.021853 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:28.025862 17180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:28.025920 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:28.025939 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:28.025956 17180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:28.025971 17180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:28.031246 17180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:28.032263 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:28.033227 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:28.042098 17180 server_base.cc:1034] running on GCE node
W20260810 23:06:28.042165 17189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.042183 17187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.042546 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:28.043143 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:28.044250 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:28.045451 17180 hybrid_clock.cc:648] HybridClock initialized: now 1786403188045384 us; error 44 us; skew 500 ppm
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:28.048171 17180 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:28.049098 17180 webserver.cc:460] Webserver started at http://127.14.75.62:33089/ using document root <none> and password file <none>
I20260810 23:06:28.049763 17180 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:28.049861 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:28.050122 17180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:28.052209 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d7eec0e405e7d60db25d291a903b78ac"
server_key_iv: "61aabfccb29fe4ebc62d44abcf74488e"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.052785 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d7eec0e405e7d60db25d291a903b78ac"
server_key_iv: "61aabfccb29fe4ebc62d44abcf74488e"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.056434 17180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:06:28.058845 17197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.060108 17180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 23:06:28.060240 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d7eec0e405e7d60db25d291a903b78ac"
server_key_iv: "61aabfccb29fe4ebc62d44abcf74488e"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.060359 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:28.077106 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:28.080190 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:28.080456 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:28.087857 17180 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42399
I20260810 23:06:28.087885 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42399 every 8 connection(s)
I20260810 23:06:28.088821 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:28.090790 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17180
I20260810 23:06:28.090884 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:28.091176 14636 external_mini_cluster.cc:1428] Setting key fdc4eace2fcdfc2798770330ba115286
I20260810 23:06:28.091734 17250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:28.099007 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777: Bootstrap starting.
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:28.101570 17250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:28.102541 17250 log.cc:824] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:28.104960 17250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777: No bootstrap required, opened a new log
I20260810 23:06:28.107151 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.092692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.14.75.62:42399)
0810 23:06:28.093151 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:28.093170 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:28.093212 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:06:28.094031 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:28.094055 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:28.094469 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:28.094856 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:28.095561 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.097065 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.097980 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.098645 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.098821 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.101828 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:28.101837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:28.101857 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:28.101896 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:28.103787 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.104228 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.104234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.104240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.104380 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.104661 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.104664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.104665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.105179 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:28.105379 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:28.105628 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:06:28.105937 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":124,"threads_started":1}
I20260810 23:06:28.108935 17250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } }
I20260810 23:06:28.109192 17250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:28.109316 17250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd61c4c089e4d0ea9eb4ff8bbaf9777, State: Initialized, Role: FOLLOWER
I20260810 23:06:28.109750 17250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } }
I20260810 23:06:28.109892 17250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:28.109959 17250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:28.110078 17250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:28.111032 17250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } }
I20260810 23:06:28.111390 17250 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd61c4c089e4d0ea9eb4ff8bbaf9777; no voters:
I20260810 23:06:28.111734 17250 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:28.111858 17256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:28.112111 17256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [term 1 LEADER]: Becoming Leader. State: Replica: 2bd61c4c089e4d0ea9eb4ff8bbaf9777, State: Running, Role: LEADER
I20260810 23:06:28.112488 17256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } }
I20260810 23:06:28.113057 17250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:28.114372 17258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd61c4c089e4d0ea9eb4ff8bbaf9777. Latest consensus state: current_term: 1 leader_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } } }
I20260810 23:06:28.114526 17258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:28.114372 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd61c4c089e4d0ea9eb4ff8bbaf9777" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42399 } } }
I20260810 23:06:28.114589 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:28.114997 17267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:28.117834 17267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:28.123574 17267 catalog_manager.cc:1347] Generated new cluster ID: 16c1ca63d7614754ba4cfcaa9e94bb9b
I20260810 23:06:28.123651 17267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:28.139555 17267 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:28.140933 17267 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:28.146412 17267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd61c4c089e4d0ea9eb4ff8bbaf9777: Generated new TSK 0
I20260810 23:06:28.147230 17267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:28.180675 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:28.293854 17280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:28.294152 17280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:28.294195 17280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:28.298395 17280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:28.298451 17280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:28.298511 17280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:28.303750 17280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:28.304809 17280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:28.305871 17280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:28.315490 17285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.315490 17288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.315541 17286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:28.315644 17280 server_base.cc:1034] running on GCE node
I20260810 23:06:28.316272 17280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:28.316852 17280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:28.318006 17280 hybrid_clock.cc:648] HybridClock initialized: now 1786403188317986 us; error 31 us; skew 500 ppm
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:28.320693 17280 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:28.321578 17280 webserver.cc:460] Webserver started at http://127.14.75.1:42347/ using document root <none> and password file <none>
I20260810 23:06:28.322075 17280 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:28.322131 17280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:28.322302 17280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:28.324256 17280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "34ae3db53e5f44e8b159dc8be1a1d417"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "a2f90ee25c2600e54ee96437b3293ea2"
server_key_iv: "085aa1c7b24e0eae5c5cba6687bb9e2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.324761 17280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "34ae3db53e5f44e8b159dc8be1a1d417"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "a2f90ee25c2600e54ee96437b3293ea2"
server_key_iv: "085aa1c7b24e0eae5c5cba6687bb9e2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.328359 17280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 23:06:28.330659 17295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.331849 17280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:28.331980 17280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "34ae3db53e5f44e8b159dc8be1a1d417"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "a2f90ee25c2600e54ee96437b3293ea2"
server_key_iv: "085aa1c7b24e0eae5c5cba6687bb9e2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.332087 17280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:28.351121 17280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:28.354086 17280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:28.354312 17280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:28.354923 17280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:28.355842 17280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:28.355917 17280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.355993 17280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:28.356035 17280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.365386 17280 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:38597
I20260810 23:06:28.365432 17413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:38597 every 8 connection(s)
I20260810 23:06:28.366317 17280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:28.367347 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17280
I20260810 23:06:28.367465 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:28.367790 14636 external_mini_cluster.cc:1428] Setting key 88d324c8760c2acf64c34e1d99031488
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:28.379873 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.368233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:35261 (local address 127.14.75.62:42399)
0810 23:06:28.368401 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:28.368406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:28.369588 (+ 1182us) server_negotiation.cc:407] Connection header received
0810 23:06:28.370511 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:28.370514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:28.370567 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:28.370648 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:28.371802 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.372341 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.373562 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.373808 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.373927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.376407 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:28.376416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:28.376418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:28.376442 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:28.377917 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.378388 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.378390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.378391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.378465 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.378774 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.378777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.378778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.379036 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:28.379121 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:28.379585 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 23:06:28.379719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:06:28.381022 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.368682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42399 (local address 127.14.75.1:35261)
0810 23:06:28.369442 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:28.369496 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:28.370306 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:28.370814 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:28.370821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:28.371170 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:28.371646 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.371659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.372474 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.372479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.373350 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.373359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.373970 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.373974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.374143 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.374791 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:28.374806 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:28.376241 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:28.378022 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.378027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.378040 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.378297 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.378545 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.378547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.378548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.378677 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.379130 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:28.379135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:28.379368 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 23:06:28.379520 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:06:28.379922 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":145,"threads_started":1}
I20260810 23:06:28.382364 17414 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42399
I20260810 23:06:28.382614 17414 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:28.383102 17414 heartbeater.cc:511] Master 127.14.75.62:42399 requested a full tablet report, sending...
I20260810 23:06:28.384701 17214 ts_manager.cc:194] Registered new tserver with Master: 34ae3db53e5f44e8b159dc8be1a1d417 (127.14.75.1:38597)
I20260810 23:06:28.385917 17214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:35261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:28.396206 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:28.508095 17419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:28.508378 17419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:28.508459 17419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:28.512528 17419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:28.512609 17419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:28.512737 17419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:28.518484 17419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:28.519570 17419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:28.520609 17419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:28.530544 17426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:28.530719 17419 server_base.cc:1034] running on GCE node
W20260810 23:06:28.530526 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.530526 17425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:28.531387 17419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:28.532145 17419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:28.533346 17419 hybrid_clock.cc:648] HybridClock initialized: now 1786403188533329 us; error 40 us; skew 500 ppm
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:28.536286 17419 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:28.537276 17419 webserver.cc:460] Webserver started at http://127.14.75.2:43411/ using document root <none> and password file <none>
I20260810 23:06:28.537909 17419 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:28.537976 17419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:28.538218 17419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:28.540314 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "40de846a5d834b4aa9a55fce21358fac"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "13c69817d28d62c2d84831a4d7c87888"
server_key_iv: "6ab64dc065dd9fe8e345ecdfdf39a66b"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.540896 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "40de846a5d834b4aa9a55fce21358fac"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "13c69817d28d62c2d84831a4d7c87888"
server_key_iv: "6ab64dc065dd9fe8e345ecdfdf39a66b"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.544777 17419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:28.547200 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.548421 17419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:28.548520 17419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "40de846a5d834b4aa9a55fce21358fac"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "13c69817d28d62c2d84831a4d7c87888"
server_key_iv: "6ab64dc065dd9fe8e345ecdfdf39a66b"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.548609 17419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:28.569144 17419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:28.572381 17419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:28.572546 17419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:28.573107 17419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:28.573982 17419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:28.574029 17419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.574070 17419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:28.574101 17419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.583050 17419 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:34603
I20260810 23:06:28.583187 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:34603 every 8 connection(s)
I20260810 23:06:28.583969 17419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:28.593456 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17419
I20260810 23:06:28.593547 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:28.593840 14636 external_mini_cluster.cc:1428] Setting key 39ecb23df8a748e8f2621b8efde252a2
I20260810 23:06:28.597945 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.585787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:42005 (local address 127.14.75.62:42399)
0810 23:06:28.585953 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:28.585958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:28.587010 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 23:06:28.587849 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:28.587852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:28.587899 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:28.587979 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:28.589130 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.589753 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.590856 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.591069 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.591177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.593764 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:28.593767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:28.593770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:28.593807 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:28.595525 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.596169 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.596171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.596172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.596257 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.596657 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.596659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.596660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.596840 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:28.596916 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:28.597643 (+ 727us) server_negotiation.cc:299] Negotiation successful
0810 23:06:28.597786 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:06:28.598810 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.586201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42399 (local address 127.14.75.2:42005)
0810 23:06:28.586865 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:28.586907 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:28.587647 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:28.588135 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:28.588141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:28.588489 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:28.588966 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.588979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.589861 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.589863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.590743 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.590752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.591224 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.591226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.591361 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.591955 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:28.591971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:28.593551 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:28.595645 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.595651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.595668 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.596061 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.596348 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.596354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.596359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.596562 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.596943 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:28.596953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:28.597330 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 23:06:28.597596 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 23:06:28.597942 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":112,"threads_started":1}
I20260810 23:06:28.600184 17556 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42399
I20260810 23:06:28.600441 17556 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:28.600945 17556 heartbeater.cc:511] Master 127.14.75.62:42399 requested a full tablet report, sending...
I20260810 23:06:28.602072 17214 ts_manager.cc:194] Registered new tserver with Master: 40de846a5d834b4aa9a55fce21358fac (127.14.75.2:34603)
I20260810 23:06:28.602905 17214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:42005
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:28.623939 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:28.736286 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:28.736572 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:28.736655 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:28.740690 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:28.740751 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:28.740808 17561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:28.746160 17561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:28.747188 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:28.748157 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:28.757632 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.757668 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:28.757747 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:28.757891 17561 server_base.cc:1034] running on GCE node
I20260810 23:06:28.758342 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:28.758970 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:28.760155 17561 hybrid_clock.cc:648] HybridClock initialized: now 1786403188760137 us; error 29 us; skew 500 ppm
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:28.762892 17561 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:28.763844 17561 webserver.cc:460] Webserver started at http://127.14.75.3:45051/ using document root <none> and password file <none>
I20260810 23:06:28.764454 17561 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:28.764547 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:28.764781 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:28.766880 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "bbf42f7600a341d7a058bab8fcd7d182"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d721c1a63b4e7b2e38e55be2bfe13bca"
server_key_iv: "3bad6cad60f12a184fdbbf11a5ed87fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.767464 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "bbf42f7600a341d7a058bab8fcd7d182"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d721c1a63b4e7b2e38e55be2bfe13bca"
server_key_iv: "3bad6cad60f12a184fdbbf11a5ed87fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.771303 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:28.773743 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.774950 17561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:06:28.775108 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "bbf42f7600a341d7a058bab8fcd7d182"
format_stamp: "Formatted at 2026-08-10 23:06:28 on dist-test-slave-srgv"
server_key: "d721c1a63b4e7b2e38e55be2bfe13bca"
server_key_iv: "3bad6cad60f12a184fdbbf11a5ed87fd"
server_key_version: "encryptionkey@0"
I20260810 23:06:28.775230 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:28.786229 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:28.789130 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:28.789350 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:28.790011 17561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:28.790864 17561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:28.790940 17561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.791013 17561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:28.791064 17561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:28.799911 17561 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:38739
I20260810 23:06:28.799930 17694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:38739 every 8 connection(s)
I20260810 23:06:28.800899 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403188, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:28.810768 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17561
I20260810 23:06:28.810863 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:28.811131 14636 external_mini_cluster.cc:1428] Setting key fd0beb8c1164510412cf71c895cb11e0
I20260810 23:06:28.815021 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.802721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53993 (local address 127.14.75.62:42399)
0810 23:06:28.802902 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:28.802907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:28.803933 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 23:06:28.804770 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:28.804773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:28.804820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:28.804898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:28.806148 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.806674 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.807767 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.807981 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.808092 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.810653 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:28.810656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:28.810658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:28.810683 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:28.812540 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.813324 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.813328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.813329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.813442 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:28.813916 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:28.813919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:28.813920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:28.814144 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:28.814243 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:28.814789 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 23:06:28.814903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:06:28.815936 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:28.803119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42399 (local address 127.14.75.3:53993)
0810 23:06:28.803784 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:28.803826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:28.804590 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:28.805061 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:28.805067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:28.805510 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:28.806009 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.806023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.806784 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.806786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.807656 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:28.807665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:28.808117 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:28.808119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:28.808271 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:28.808907 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:28.808923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:28.810437 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:28.812765 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.812772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.812790 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.813204 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.813542 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:28.813546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:28.813548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:28.813817 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:28.814275 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:28.814280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:28.814535 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 23:06:28.814718 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 23:06:28.815090 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":120,"threads_started":1}
I20260810 23:06:28.817068 17695 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42399
I20260810 23:06:28.817323 17695 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:28.817845 17695 heartbeater.cc:511] Master 127.14.75.62:42399 requested a full tablet report, sending...
I20260810 23:06:28.818879 17214 ts_manager.cc:194] Registered new tserver with Master: bbf42f7600a341d7a058bab8fcd7d182 (127.14.75.3:38739)
I20260810 23:06:28.819417 17214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:53993
I20260810 23:06:28.825366 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:06:28.839105 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:28.837726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.14.75.62:42399)
0810 23:06:28.837856 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:28.837859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:28.837987 (+ 128us) server_negotiation.cc:407] Connection header received
0810 23:06:28.838085 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:28.838088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:28.838132 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:28.838223 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:28.838975 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60728: BlockingRecv error: recv got EOF from 127.0.0.1:60728 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:06:28.840286 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17280
I20260810 23:06:28.846498 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17419
I20260810 23:06:28.852982 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17561
I20260810 23:06:28.858906 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17180
2026-08-10T23:06:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2147 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17709](info): set up 2 sockets
Aug 10 23:06:28 dist-test-slave-srgv krb5kdc[17709](info): commencing operation
krb5kdc: starting...
W20260810 23:06:29.922508 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:30.051730 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36797
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:36797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:30.163707 17723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:30.164078 17723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:30.164296 17723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:30.168447 17723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:30.168557 17723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:30.168599 17723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:30.168637 17723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:30.168670 17723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:30.173918 17723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:36797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:30.175073 17723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:30.176119 17723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:30.184854 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:30.184854 17729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:30.184897 17731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:30.185217 17723 server_base.cc:1034] running on GCE node
I20260810 23:06:30.185943 17723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:30.187101 17723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:30.188297 17723 hybrid_clock.cc:648] HybridClock initialized: now 1786403190188256 us; error 56 us; skew 500 ppm
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403190, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:30.195113 17723 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:30.196211 17723 webserver.cc:460] Webserver started at http://127.14.75.62:36547/ using document root <none> and password file <none>
I20260810 23:06:30.196825 17723 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:30.196929 17723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:30.197180 17723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:30.199440 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "34b0f85128d9472f8973e73a823fee27"
format_stamp: "Formatted at 2026-08-10 23:06:30 on dist-test-slave-srgv"
server_key: "0bb9426f6ec9789ce782852e6a3a859e"
server_key_iv: "5b512cab45e0fc022a1a4cada82a0b15"
server_key_version: "encryptionkey@0"
I20260810 23:06:30.200063 17723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "34b0f85128d9472f8973e73a823fee27"
format_stamp: "Formatted at 2026-08-10 23:06:30 on dist-test-slave-srgv"
server_key: "0bb9426f6ec9789ce782852e6a3a859e"
server_key_iv: "5b512cab45e0fc022a1a4cada82a0b15"
server_key_version: "encryptionkey@0"
I20260810 23:06:30.204174 17723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:30.206818 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:30.208072 17723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:30.208233 17723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "34b0f85128d9472f8973e73a823fee27"
format_stamp: "Formatted at 2026-08-10 23:06:30 on dist-test-slave-srgv"
server_key: "0bb9426f6ec9789ce782852e6a3a859e"
server_key_iv: "5b512cab45e0fc022a1a4cada82a0b15"
server_key_version: "encryptionkey@0"
I20260810 23:06:30.208361 17723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:30.240893 17723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:30.244115 17723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:30.244355 17723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:30.251598 17723 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:36797
I20260810 23:06:30.251935 17795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:36797 every 8 connection(s)
I20260810 23:06:30.252651 17723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:30.255474 17796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:30.258710 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17723
I20260810 23:06:30.258812 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:30.259081 14636 external_mini_cluster.cc:1428] Setting key 2193684544e352b6cda8af044010afb4
I20260810 23:06:30.261601 17796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: Bootstrap starting.
I20260810 23:06:30.264189 17796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:30.265100 17796 log.cc:824] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:30.267423 17796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: No bootstrap required, opened a new log
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:30.271190 17796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } }
I20260810 23:06:30.271466 17796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:30.271543 17796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b0f85128d9472f8973e73a823fee27, State: Initialized, Role: FOLLOWER
I20260810 23:06:30.271991 17796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } }
I20260810 23:06:30.272192 17796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:30.272285 17796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:30.272399 17796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:30.273460 17796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } }
I20260810 23:06:30.273801 17796 leader_election.cc:302] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b0f85128d9472f8973e73a823fee27; no voters:
I20260810 23:06:30.274163 17796 leader_election.cc:288] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:30.274243 17802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:30.274370 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:30.260545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.14.75.62:36797)
0810 23:06:30.261004 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:30.261019 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:30.261056 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:30.261784 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:30.261804 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:30.262135 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:30.262421 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:30.263409 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:30.264453 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:30.265639 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:30.266232 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:30.266353 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:30.269161 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:30.269175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:30.269188 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:30.269215 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:30.270978 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:30.271446 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:30.271451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:30.271455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:30.271634 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:30.271979 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:30.271984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:30.271986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:30.272549 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:30.272771 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:30.273084 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:06:30.273379 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":162,"threads_started":1}
I20260810 23:06:30.274462 17802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [term 1 LEADER]: Becoming Leader. State: Replica: 34b0f85128d9472f8973e73a823fee27, State: Running, Role: LEADER
I20260810 23:06:30.274978 17802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } }
I20260810 23:06:30.275485 17796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:30.276188 17802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b0f85128d9472f8973e73a823fee27. Latest consensus state: current_term: 1 leader_uuid: "34b0f85128d9472f8973e73a823fee27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } } }
I20260810 23:06:30.277004 17802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:30.276887 17803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b0f85128d9472f8973e73a823fee27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0f85128d9472f8973e73a823fee27" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36797 } } }
I20260810 23:06:30.277321 17803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:30.277599 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 23:06:30.280598 17818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:30.280691 17818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:30.280838 17811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:30.281752 17811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:30.287241 17811 catalog_manager.cc:1347] Generated new cluster ID: 8fa9c2596740435d8296fdbb67b837a7
I20260810 23:06:30.287307 17811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:30.296303 17811 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:30.297367 17811 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:30.301465 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:30.295611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.14.75.62:36797)
0810 23:06:30.295714 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:30.295719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:30.295975 (+ 256us) server_negotiation.cc:407] Connection header received
0810 23:06:30.296176 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:30.296181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:30.296235 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:30.296290 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:30.297041 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:30.297783 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:30.298991 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:30.299356 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:30.299452 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:30.300412 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:30.300416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:30.300419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:30.300956 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:30.300987 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:30.301004 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:30.301088 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:30.301210 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:06:30.301296 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:06:30.315821 17811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b0f85128d9472f8973e73a823fee27: Generated new TSK 0
I20260810 23:06:30.316627 17811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:30.328857 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17723
2026-08-10T23:06:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1468 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17831](info): set up 2 sockets
Aug 10 23:06:30 dist-test-slave-srgv krb5kdc[17831](info): commencing operation
krb5kdc: starting...
W20260810 23:06:31.430230 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:31 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:31.556875 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46737
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:31.668679 17846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:31.669021 17846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:31.669101 17846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:31.673105 17846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:31.673218 17846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:31.673252 17846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:31.673292 17846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:31.673324 17846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:31.678840 17846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:31.679935 17846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:31.680981 17846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:31.689630 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:31.689672 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:31.689634 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:31.690084 17846 server_base.cc:1034] running on GCE node
I20260810 23:06:31.690762 17846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:31.692028 17846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:31.693187 17846 hybrid_clock.cc:648] HybridClock initialized: now 1786403191693169 us; error 32 us; skew 500 ppm
Aug 10 23:06:31 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:31.695971 17846 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:31.696883 17846 webserver.cc:460] Webserver started at http://127.14.75.62:46757/ using document root <none> and password file <none>
I20260810 23:06:31.697510 17846 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:31.697613 17846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:31.697847 17846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:31.699918 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "256867a3f7d849d1936ce39783763550"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "9cef3a83305e108e5f2cd69d1f86dd4b"
server_key_iv: "313a636d9c7ee24d1081c91a2ed66b01"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.700495 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "256867a3f7d849d1936ce39783763550"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "9cef3a83305e108e5f2cd69d1f86dd4b"
server_key_iv: "313a636d9c7ee24d1081c91a2ed66b01"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.704277 17846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:06:31.706744 17863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:31.708120 17846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:31.708236 17846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "256867a3f7d849d1936ce39783763550"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "9cef3a83305e108e5f2cd69d1f86dd4b"
server_key_iv: "313a636d9c7ee24d1081c91a2ed66b01"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.708325 17846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:31.720075 17846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:31.723644 17846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:31.723806 17846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:31.730880 17846 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46737
I20260810 23:06:31.731222 17919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46737 every 8 connection(s)
I20260810 23:06:31.731791 17846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:31.733680 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17846
I20260810 23:06:31.733785 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:31.734077 14636 external_mini_cluster.cc:1428] Setting key b6c510a91a743aa47506fcb735acf761
I20260810 23:06:31.734844 17920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:31.742019 17920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550: Bootstrap starting.
Aug 10 23:06:31 dist-test-slave-srgv krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:31.744827 17920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:31.745841 17920 log.cc:824] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:31.748250 17920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550: No bootstrap required, opened a new log
I20260810 23:06:31.749927 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:31.735555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.14.75.62:46737)
0810 23:06:31.735992 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:31.736011 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:31.736065 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:06:31.736933 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:31.736958 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:31.737393 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:31.737889 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:31.738597 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:31.740083 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:31.740935 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:31.741628 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:31.741783 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:31.744479 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:31.744485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:31.744502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:31.744536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:31.746651 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:31.747185 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:31.747191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:31.747198 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:31.747336 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:31.747616 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:31.747619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:31.747621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:31.748122 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:31.748304 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:31.748626 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:06:31.748925 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260810 23:06:31.752575 17920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } }
I20260810 23:06:31.752794 17920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:31.752830 17920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256867a3f7d849d1936ce39783763550, State: Initialized, Role: FOLLOWER
I20260810 23:06:31.753254 17920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } }
I20260810 23:06:31.753398 17920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:31.753511 17920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:31.753590 17920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:31.754504 17920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } }
I20260810 23:06:31.754796 17920 leader_election.cc:302] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256867a3f7d849d1936ce39783763550; no voters:
I20260810 23:06:31.755091 17920 leader_election.cc:288] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:31.755239 17926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:31.755532 17926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [term 1 LEADER]: Becoming Leader. State: Replica: 256867a3f7d849d1936ce39783763550, State: Running, Role: LEADER
I20260810 23:06:31.755864 17926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } }
I20260810 23:06:31.756367 17920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:31.757630 17928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256867a3f7d849d1936ce39783763550. Latest consensus state: current_term: 1 leader_uuid: "256867a3f7d849d1936ce39783763550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } } }
I20260810 23:06:31.757647 17927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256867a3f7d849d1936ce39783763550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256867a3f7d849d1936ce39783763550" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46737 } } }
I20260810 23:06:31.757850 17928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:31.757880 17927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:31.758399 17935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:31.761771 17935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:31.767635 17935 catalog_manager.cc:1347] Generated new cluster ID: 7e4eb8dd94fd419b8ae60e1afb002a91
I20260810 23:06:31.767729 17935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:31.778523 17935 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:31.779667 17935 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:31.789177 17935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 256867a3f7d849d1936ce39783763550: Generated new TSK 0
I20260810 23:06:31.789987 17935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:31.828250 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:31.943996 17950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:31.944471 17950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:31.944623 17950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:31.951052 17950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:31.951124 17950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:31.951232 17950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:31.956880 17950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:31.958189 17950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:31.959304 17950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:31.969799 17955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:31.969801 17958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:31.969799 17956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:31.970237 17950 server_base.cc:1034] running on GCE node
I20260810 23:06:31.970621 17950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:31.971289 17950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:31.972458 17950 hybrid_clock.cc:648] HybridClock initialized: now 1786403191972424 us; error 47 us; skew 500 ppm
Aug 10 23:06:31 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:31.975330 17950 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:31.976302 17950 webserver.cc:460] Webserver started at http://127.14.75.1:35111/ using document root <none> and password file <none>
I20260810 23:06:31.976879 17950 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:31.976974 17950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:31.977244 17950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:31.979353 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "925a25f30c044be99bc67822e437f37e"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "8a5503b6360bb5b299fad497776ad3db"
server_key_iv: "4e0129c868cfc7531f51b393a9dc40ab"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.979936 17950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "925a25f30c044be99bc67822e437f37e"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "8a5503b6360bb5b299fad497776ad3db"
server_key_iv: "4e0129c868cfc7531f51b393a9dc40ab"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.983924 17950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 23:06:31.986596 17966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:31.987811 17950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:31.987959 17950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "925a25f30c044be99bc67822e437f37e"
format_stamp: "Formatted at 2026-08-10 23:06:31 on dist-test-slave-srgv"
server_key: "8a5503b6360bb5b299fad497776ad3db"
server_key_iv: "4e0129c868cfc7531f51b393a9dc40ab"
server_key_version: "encryptionkey@0"
I20260810 23:06:31.988085 17950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:31.998785 17950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:32.001842 17950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:32.002060 17950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:32.002677 17950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:32.003532 17950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:32.003609 17950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.003685 17950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:32.003741 17950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.012804 17950 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:45557
I20260810 23:06:32.012876 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:45557 every 8 connection(s)
I20260810 23:06:32.013794 17950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:32.015770 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 17950
I20260810 23:06:32.015893 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:32.016191 14636 external_mini_cluster.cc:1428] Setting key a07f299c1c219f98b3d0febd5d40f9f1
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403191, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:32.029340 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.015586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:38911 (local address 127.14.75.62:46737)
0810 23:06:32.015764 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:32.015769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:32.016741 (+ 972us) server_negotiation.cc:407] Connection header received
0810 23:06:32.018406 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:32.018409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:32.018452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:32.018520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:32.020100 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.020968 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.022216 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.022646 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.022776 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.025604 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:32.025610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:32.025613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:32.025648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:32.027380 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.027918 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.027920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.027921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.027996 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.028331 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.028333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.028334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.028507 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:32.028580 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:32.029090 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 23:06:32.029209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:32.030544 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.016030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46737 (local address 127.14.75.1:38911)
0810 23:06:32.016572 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:32.016635 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:32.018160 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:32.018734 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:32.018747 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:32.019297 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:32.019944 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.019959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.021096 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.021098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.022080 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.022090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.022811 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.022814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.022958 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.023621 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:32.023639 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:32.025353 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:32.027486 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.027491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.027503 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.027771 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.028095 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.028098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.028100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.028241 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.028593 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:32.028598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:32.028849 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 23:06:32.029038 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 23:06:32.029409 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260810 23:06:32.031980 18087 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46737
I20260810 23:06:32.032231 18087 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:32.032742 18087 heartbeater.cc:511] Master 127.14.75.62:46737 requested a full tablet report, sending...
I20260810 23:06:32.034521 17883 ts_manager.cc:194] Registered new tserver with Master: 925a25f30c044be99bc67822e437f37e (127.14.75.1:45557)
I20260810 23:06:32.036237 17883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:38911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:32.048906 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:32.164040 18092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:32.164342 18092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:32.164413 18092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:32.168612 18092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:32.168687 18092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:32.168785 18092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:32.174300 18092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:32.175526 18092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:32.176611 18092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:32.186265 18100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:32.186308 18097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:32.186265 18098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:32.186666 18092 server_base.cc:1034] running on GCE node
I20260810 23:06:32.187088 18092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:32.187733 18092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:32.188916 18092 hybrid_clock.cc:648] HybridClock initialized: now 1786403192188886 us; error 54 us; skew 500 ppm
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:32.191895 18092 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:32.192891 18092 webserver.cc:460] Webserver started at http://127.14.75.2:41699/ using document root <none> and password file <none>
I20260810 23:06:32.193521 18092 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:32.193619 18092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:32.193859 18092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:32.195945 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "58ecf44572984e09acff904044ac2575"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "00133b1c9bdd4c312b50ce1ed8d6fb20"
server_key_iv: "cff6bf71a9cd605d72c9018ea3b484dc"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.196524 18092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "58ecf44572984e09acff904044ac2575"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "00133b1c9bdd4c312b50ce1ed8d6fb20"
server_key_iv: "cff6bf71a9cd605d72c9018ea3b484dc"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.200268 18092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:06:32.202811 18107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.204030 18092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:32.204170 18092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "58ecf44572984e09acff904044ac2575"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "00133b1c9bdd4c312b50ce1ed8d6fb20"
server_key_iv: "cff6bf71a9cd605d72c9018ea3b484dc"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.204286 18092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:32.224998 18092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:32.228550 18092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:32.228783 18092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:32.229398 18092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:32.230290 18092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:32.230374 18092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.230445 18092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:32.230496 18092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.239645 18092 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37521
I20260810 23:06:32.239663 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37521 every 8 connection(s)
I20260810 23:06:32.240548 18092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:32.246261 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18092
I20260810 23:06:32.246428 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:32.246727 14636 external_mini_cluster.cc:1428] Setting key 2a391136b1f7661b017ae434f2fcd10a
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:32.255857 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.242377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:58495 (local address 127.14.75.62:46737)
0810 23:06:32.242524 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:32.242528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:32.243310 (+ 782us) server_negotiation.cc:407] Connection header received
0810 23:06:32.244710 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:32.244713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:32.244770 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:32.244886 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:32.246181 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.246860 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.248135 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.248465 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.248588 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.252116 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:32.252119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:32.252121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:32.252144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:32.253724 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.254279 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.254283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.254287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.254367 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.254723 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.254726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.254728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.254916 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:32.255038 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:32.255544 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 23:06:32.255679 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:32.256773 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.242721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46737 (local address 127.14.75.2:58495)
0810 23:06:32.243154 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:32.243200 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:32.244491 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:32.245059 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:32.245082 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:32.245516 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:32.246002 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.246037 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.246996 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.247004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.248010 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.248022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.248611 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.248614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.248761 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.250208 (+ 1447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:32.250237 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:32.251938 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:32.253846 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.253851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.253867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.254129 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.254464 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.254468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.254470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.254618 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.255039 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:32.255048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:32.255311 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 23:06:32.255504 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 23:06:32.255841 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260810 23:06:32.258154 18226 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46737
I20260810 23:06:32.258399 18226 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:32.258934 18226 heartbeater.cc:511] Master 127.14.75.62:46737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:06:32.260042 17883 ts_manager.cc:194] Registered new tserver with Master: 58ecf44572984e09acff904044ac2575 (127.14.75.2:37521)
I20260810 23:06:32.260627 17883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:58495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:32.276414 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:32.388972 18231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:32.389252 18231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:32.389297 18231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:32.393617 18231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:32.393688 18231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:32.393750 18231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:32.399104 18231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:32.400166 18231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:32.401121 18231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:32.411072 18239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:32.411062 18236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:32.411084 18237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:32.411302 18231 server_base.cc:1034] running on GCE node
I20260810 23:06:32.411834 18231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:32.412549 18231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:32.413735 18231 hybrid_clock.cc:648] HybridClock initialized: now 1786403192413701 us; error 47 us; skew 500 ppm
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:32.416512 18231 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:32.417485 18231 webserver.cc:460] Webserver started at http://127.14.75.3:44123/ using document root <none> and password file <none>
I20260810 23:06:32.418071 18231 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:32.418167 18231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:32.418408 18231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:32.420492 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "9250e2a66b474e1bad2fd79683b46bd3"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "65e88663b45e0e49f497cf413aed127c"
server_key_iv: "57a6def9c0efc5c8fe5826f3ccc55bef"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.421075 18231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "9250e2a66b474e1bad2fd79683b46bd3"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "65e88663b45e0e49f497cf413aed127c"
server_key_iv: "57a6def9c0efc5c8fe5826f3ccc55bef"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.425019 18231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 23:06:32.427464 18246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.428643 18231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:32.428794 18231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "9250e2a66b474e1bad2fd79683b46bd3"
format_stamp: "Formatted at 2026-08-10 23:06:32 on dist-test-slave-srgv"
server_key: "65e88663b45e0e49f497cf413aed127c"
server_key_iv: "57a6def9c0efc5c8fe5826f3ccc55bef"
server_key_version: "encryptionkey@0"
I20260810 23:06:32.428910 18231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:32.445163 18231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:32.448163 18231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:32.448397 18231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:32.449038 18231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:32.449944 18231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:32.450019 18231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.450093 18231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:32.450148 18231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:32.459555 18231 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:34759
I20260810 23:06:32.459621 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:34759 every 8 connection(s)
I20260810 23:06:32.460527 18231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:32.463842 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18231
I20260810 23:06:32.463941 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:32.464244 14636 external_mini_cluster.cc:1428] Setting key 4fc2ac499e742463debde56b10c73856
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:32.474785 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.462443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:34003 (local address 127.14.75.62:46737)
0810 23:06:32.462599 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:32.462602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:32.463429 (+ 827us) server_negotiation.cc:407] Connection header received
0810 23:06:32.465043 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:32.465046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:32.465124 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:32.465210 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:32.466771 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.467297 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.468417 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.468633 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.468719 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.471259 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:32.471262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:32.471264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:32.471288 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:32.472561 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.473247 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.473249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.473250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.473330 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.473766 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.473770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.473771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.473968 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:32.474050 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:32.474537 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 23:06:32.474649 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:06:32.475606 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.462821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46737 (local address 127.14.75.3:34003)
0810 23:06:32.463274 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:32.463317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:32.464824 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:32.465391 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:32.465398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:32.465963 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:32.466642 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.466658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.467408 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.467410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.468312 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:32.468321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.468766 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.468768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:32.468915 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.469583 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:32.469600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:32.471096 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:32.472683 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.472689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.472702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.473147 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.473459 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:32.473463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:32.473465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:32.473659 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:32.474060 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:32.474065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:32.474318 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:06:32.474474 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 23:06:32.474740 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260810 23:06:32.476799 18365 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46737
I20260810 23:06:32.477053 18365 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:32.477610 18365 heartbeater.cc:511] Master 127.14.75.62:46737 requested a full tablet report, sending...
I20260810 23:06:32.478549 17883 ts_manager.cc:194] Registered new tserver with Master: 9250e2a66b474e1bad2fd79683b46bd3 (127.14.75.3:34759)
I20260810 23:06:32.479121 17883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:34003
I20260810 23:06:32.490531 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403192, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:32.517457 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:32.508764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.14.75.62:46737)
0810 23:06:32.508965 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:32.508971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:32.509307 (+ 336us) server_negotiation.cc:407] Connection header received
0810 23:06:32.509445 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:32.509450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:32.509506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:32.509587 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:32.510298 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.510808 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.511683 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:32.511887 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:32.511964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:32.514511 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:32.514514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:32.514516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:32.514540 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:32.515993 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.516374 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.516376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.516377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.516450 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:32.516668 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:32.516670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:32.516671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:32.516888 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:32.516986 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:32.517184 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:06:32.517288 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 23:06:32.517858 17883 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35370
I20260810 23:06:32.518703 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17950
I20260810 23:06:32.525098 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18092
I20260810 23:06:32.531109 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18231
I20260810 23:06:32.538327 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 17846
2026-08-10T23:06:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2211 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[18377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[18377](info): set up 2 sockets
Aug 10 23:06:32 dist-test-slave-srgv krb5kdc[18377](info): commencing operation
krb5kdc: starting...
W20260810 23:06:33.606877 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:33 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:33.733101 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40665
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:40665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:33.845578 18390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:33.845916 18390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:33.845959 18390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:33.850082 18390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:33.850148 18390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:33.850168 18390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:33.850185 18390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:33.850245 18390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:33.855718 18390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:40665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:33.856877 18390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:33.857980 18390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:33.867237 18396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:33.867254 18395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:33.867447 18390 server_base.cc:1034] running on GCE node
W20260810 23:06:33.867254 18398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:33.868108 18390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:33.869304 18390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:33.870494 18390 hybrid_clock.cc:648] HybridClock initialized: now 1786403193870487 us; error 32 us; skew 500 ppm
Aug 10 23:06:33 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403193, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:33.873368 18390 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:33.874333 18390 webserver.cc:460] Webserver started at http://127.14.75.62:44145/ using document root <none> and password file <none>
I20260810 23:06:33.874907 18390 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:33.874971 18390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:33.875195 18390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:33.877312 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d"
format_stamp: "Formatted at 2026-08-10 23:06:33 on dist-test-slave-srgv"
server_key: "9b8883a4a2a62de8d3acc730bcf9bd23"
server_key_iv: "d642f95fe6c092acaff108b2d4c7bf50"
server_key_version: "encryptionkey@0"
I20260810 23:06:33.877964 18390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d"
format_stamp: "Formatted at 2026-08-10 23:06:33 on dist-test-slave-srgv"
server_key: "9b8883a4a2a62de8d3acc730bcf9bd23"
server_key_iv: "d642f95fe6c092acaff108b2d4c7bf50"
server_key_version: "encryptionkey@0"
I20260810 23:06:33.881861 18390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:06:33.884370 18405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:33.885641 18390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:33.885792 18390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d"
format_stamp: "Formatted at 2026-08-10 23:06:33 on dist-test-slave-srgv"
server_key: "9b8883a4a2a62de8d3acc730bcf9bd23"
server_key_iv: "d642f95fe6c092acaff108b2d4c7bf50"
server_key_version: "encryptionkey@0"
I20260810 23:06:33.885913 18390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:33.897984 18390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:33.901225 18390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:33.901480 18390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:33.908839 18390 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:40665
I20260810 23:06:33.908869 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:40665 every 8 connection(s)
I20260810 23:06:33.909915 18390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:33.912770 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:33.918303 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d: Bootstrap starting.
I20260810 23:06:33.920238 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18390
I20260810 23:06:33.920369 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:33.920656 14636 external_mini_cluster.cc:1428] Setting key b1a2a98e888c07c2f986ed1a96d39709
I20260810 23:06:33.921096 18463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:33.922075 18463 log.cc:824] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:33.923985 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d: No bootstrap required, opened a new log
I20260810 23:06:33.927469 18463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } }
I20260810 23:06:33.927811 18463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:33.927886 18463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857c7c0bccac4bc7a9f11bbd3d0e2f6d, State: Initialized, Role: FOLLOWER
I20260810 23:06:33.928385 18463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } }
I20260810 23:06:33.928542 18463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:33.928632 18463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:33.928740 18463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:06:33 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:33.930056 18463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } }
I20260810 23:06:33.930414 18463 leader_election.cc:302] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857c7c0bccac4bc7a9f11bbd3d0e2f6d; no voters:
I20260810 23:06:33.930759 18463 leader_election.cc:288] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:33.930879 18468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:33.931214 18468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [term 1 LEADER]: Becoming Leader. State: Replica: 857c7c0bccac4bc7a9f11bbd3d0e2f6d, State: Running, Role: LEADER
I20260810 23:06:33.931602 18468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } }
I20260810 23:06:33.932097 18463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:33.933307 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } } }
I20260810 23:06:33.933779 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:33.933646 18469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857c7c0bccac4bc7a9f11bbd3d0e2f6d. Latest consensus state: current_term: 1 leader_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857c7c0bccac4bc7a9f11bbd3d0e2f6d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40665 } } }
I20260810 23:06:33.934098 18469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:33.934393 18480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:33.936352 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:33.921921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.14.75.62:40665)
0810 23:06:33.922421 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:33.922438 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:33.922474 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:06:33.923119 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:33.923134 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:33.923449 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:33.923800 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:33.924658 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:33.925686 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:33.926553 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:33.927030 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:33.927205 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:33.930313 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:33.930322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:33.930343 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:33.930371 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:33.932550 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:33.933152 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:33.933160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:33.933167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:33.933329 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:33.933704 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:33.933710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:33.933712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:33.934286 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:33.934488 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:33.935369 (+ 881us) server_negotiation.cc:299] Negotiation successful
0810 23:06:33.935612 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":159,"threads_started":1}
I20260810 23:06:33.938311 18480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:33.945740 18480 catalog_manager.cc:1347] Generated new cluster ID: 6096c9759de24b89a4b615d8cb9ce300
I20260810 23:06:33.945894 18480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:33.961941 18480 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:33.962962 18480 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:33.967085 18480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857c7c0bccac4bc7a9f11bbd3d0e2f6d: Generated new TSK 0
I20260810 23:06:33.967787 18480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:34.000906 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:34.128451 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:34.128741 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:34.128786 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:34.133435 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:34.133528 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:34.133648 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:34.139252 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:34.140413 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:34.141507 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:34.151264 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.151328 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.151296 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:34.151549 18489 server_base.cc:1034] running on GCE node
I20260810 23:06:34.152079 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:34.152730 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:34.153937 18489 hybrid_clock.cc:648] HybridClock initialized: now 1786403194153902 us; error 48 us; skew 500 ppm
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:34.156786 18489 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:34.157862 18489 webserver.cc:460] Webserver started at http://127.14.75.1:34043/ using document root <none> and password file <none>
I20260810 23:06:34.158439 18489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:34.158506 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:34.158732 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:34.160823 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "4dc3e940a05b41d8a6b8cab5f69caa98"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "ac69722679d733bdeccca1032792f9f8"
server_key_iv: "364db3d84ab6264c6760bc8ac8e5271d"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.161521 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "4dc3e940a05b41d8a6b8cab5f69caa98"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "ac69722679d733bdeccca1032792f9f8"
server_key_iv: "364db3d84ab6264c6760bc8ac8e5271d"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.165334 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:06:34.167899 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.169142 18489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:34.169271 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "4dc3e940a05b41d8a6b8cab5f69caa98"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "ac69722679d733bdeccca1032792f9f8"
server_key_iv: "364db3d84ab6264c6760bc8ac8e5271d"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.169382 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:34.185750 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:34.188722 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:34.188952 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:34.189692 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:34.190593 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:34.190642 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.190721 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:34.190753 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.200101 18489 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:36633
I20260810 23:06:34.200129 18621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:36633 every 8 connection(s)
I20260810 23:06:34.201103 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:34.209641 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18489
I20260810 23:06:34.209789 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:34.210119 14636 external_mini_cluster.cc:1428] Setting key 8643580c53fd1997c6e68b290db8d3d2
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:34.215456 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.203012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:45827 (local address 127.14.75.62:40665)
0810 23:06:34.203161 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:34.203165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:34.203996 (+ 831us) server_negotiation.cc:407] Connection header received
0810 23:06:34.205354 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:34.205356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:34.205456 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:34.205542 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:34.206704 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.207251 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.208346 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.208577 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.208655 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.211519 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:34.211522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:34.211524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:34.211548 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:34.213262 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.213854 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.213858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.213860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.213943 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.214314 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.214319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.214320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.214520 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:34.214627 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:34.215146 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:06:34.215284 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:06:34.216428 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.203387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40665 (local address 127.14.75.1:45827)
0810 23:06:34.203838 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:34.203882 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:34.205156 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:34.205716 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:34.205723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:34.206083 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:34.206559 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.206572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.207363 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.207365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.208235 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.208243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.208716 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.208719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.208857 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.209471 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:34.209496 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:34.211338 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:34.213392 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.213399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.213441 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.213739 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.214041 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.214044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.214046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.214216 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.214627 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:34.214635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:34.214905 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 23:06:34.215105 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:06:34.215468 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
I20260810 23:06:34.217808 18622 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40665
I20260810 23:06:34.218082 18622 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:34.218606 18622 heartbeater.cc:511] Master 127.14.75.62:40665 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:34.220280 18423 ts_manager.cc:194] Registered new tserver with Master: 4dc3e940a05b41d8a6b8cab5f69caa98 (127.14.75.1:36633)
I20260810 23:06:34.221505 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:45827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:34.238101 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:34.351765 18628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:34.352092 18628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:34.352165 18628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:34.356386 18628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:34.356489 18628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:34.356585 18628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:34.362295 18628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:34.363436 18628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:34.364495 18628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:34.374102 18636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.374125 18633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.374256 18634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:34.374476 18628 server_base.cc:1034] running on GCE node
I20260810 23:06:34.374982 18628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:34.375622 18628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:34.376825 18628 hybrid_clock.cc:648] HybridClock initialized: now 1786403194376809 us; error 30 us; skew 500 ppm
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:34.379773 18628 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:34.380699 18628 webserver.cc:460] Webserver started at http://127.14.75.2:38629/ using document root <none> and password file <none>
I20260810 23:06:34.381228 18628 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:34.381289 18628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:34.381518 18628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:34.383587 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "e46b70464db141f78185acfea4d8dd75"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "a04e9bfbe66a8c7a8bff993c613f4c94"
server_key_iv: "e739b7e43cf7b22f54e945c93cd76d93"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.384222 18628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "e46b70464db141f78185acfea4d8dd75"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "a04e9bfbe66a8c7a8bff993c613f4c94"
server_key_iv: "e739b7e43cf7b22f54e945c93cd76d93"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.388151 18628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:34.390755 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.391978 18628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:34.392108 18628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "e46b70464db141f78185acfea4d8dd75"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "a04e9bfbe66a8c7a8bff993c613f4c94"
server_key_iv: "e739b7e43cf7b22f54e945c93cd76d93"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.392221 18628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:34.410897 18628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:34.414546 18628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:34.414788 18628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:34.415439 18628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:34.416316 18628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:34.416368 18628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.416438 18628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:34.416484 18628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.426458 18628 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33877
I20260810 23:06:34.426483 18764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33877 every 8 connection(s)
I20260810 23:06:34.427403 18628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:34.435331 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18628
I20260810 23:06:34.435479 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:34.435828 14636 external_mini_cluster.cc:1428] Setting key 8a64b1d1cc40a650a1d5b3164b1566be
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:34.442230 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.429273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:38691 (local address 127.14.75.62:40665)
0810 23:06:34.429477 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:34.429482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:34.430387 (+ 905us) server_negotiation.cc:407] Connection header received
0810 23:06:34.431759 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:34.431762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:34.431811 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:34.431880 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:34.433213 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.433884 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.435182 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.435560 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.435652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.438523 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:34.438526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:34.438528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:34.438551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:34.440042 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.440584 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.440591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.440595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.440679 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.441030 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.441035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.441037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.441230 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:34.441342 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:34.441920 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 23:06:34.442049 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:06:34.443109 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.429745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40665 (local address 127.14.75.2:38691)
0810 23:06:34.430222 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:34.430268 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:34.431547 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:34.432045 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:34.432051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:34.432484 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:34.433045 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.433059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.434013 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.434016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.435014 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.435050 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.435736 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.435740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.435941 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.436669 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:34.436693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:34.438345 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:34.440181 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.440187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.440204 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.440477 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.440780 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.440783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.440785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.440933 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.441342 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:34.441349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:34.441679 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 23:06:34.441879 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:06:34.442248 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260810 23:06:34.444331 18765 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40665
I20260810 23:06:34.444613 18765 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:34.445261 18765 heartbeater.cc:511] Master 127.14.75.62:40665 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:34.446452 18423 ts_manager.cc:194] Registered new tserver with Master: e46b70464db141f78185acfea4d8dd75 (127.14.75.2:33877)
I20260810 23:06:34.447077 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:38691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:34.464324 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:34.577319 18770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:34.577690 18770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:34.577739 18770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:34.581913 18770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:34.581976 18770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:34.582038 18770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:34.587711 18770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:34.588798 18770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:34.589959 18770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:34.599888 18776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.599927 18779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:34.600958 18777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:34.601109 18770 server_base.cc:1034] running on GCE node
I20260810 23:06:34.601591 18770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:34.602232 18770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:34.603430 18770 hybrid_clock.cc:648] HybridClock initialized: now 1786403194603415 us; error 34 us; skew 500 ppm
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:34.606419 18770 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:34.607302 18770 webserver.cc:460] Webserver started at http://127.14.75.3:44329/ using document root <none> and password file <none>
I20260810 23:06:34.607844 18770 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:34.607903 18770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:34.608070 18770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:34.610208 18770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "912b6f14785944398be824a65a92ac4b"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "470153e3f672286a27b30df046522e52"
server_key_iv: "b5b99b922c18a4c75c9d1e64c88def10"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.610805 18770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "912b6f14785944398be824a65a92ac4b"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "470153e3f672286a27b30df046522e52"
server_key_iv: "b5b99b922c18a4c75c9d1e64c88def10"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.614657 18770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:34.617169 18787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.618510 18770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:34.618616 18770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "912b6f14785944398be824a65a92ac4b"
format_stamp: "Formatted at 2026-08-10 23:06:34 on dist-test-slave-srgv"
server_key: "470153e3f672286a27b30df046522e52"
server_key_iv: "b5b99b922c18a4c75c9d1e64c88def10"
server_key_version: "encryptionkey@0"
I20260810 23:06:34.618724 18770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:34.640129 18770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:34.643770 18770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:34.643939 18770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:34.644510 18770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:34.645321 18770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:34.645377 18770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.645467 18770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:34.645507 18770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:34.654413 18770 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:44859
I20260810 23:06:34.654444 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:44859 every 8 connection(s)
I20260810 23:06:34.655354 18770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:34.661644 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18770
I20260810 23:06:34.661815 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:34.662127 14636 external_mini_cluster.cc:1428] Setting key 6d2b79c9dc5802400d9927da6c780478
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:34.669508 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.657148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:45493 (local address 127.14.75.62:40665)
0810 23:06:34.657298 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:34.657301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:34.658164 (+ 863us) server_negotiation.cc:407] Connection header received
0810 23:06:34.659569 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:34.659572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:34.659634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:34.659717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:34.660897 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.661564 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.663064 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.663286 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.663393 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.666207 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:34.666210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:34.666212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:34.666238 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:34.667735 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.668187 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.668189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.668191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.668266 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.668551 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.668553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.668554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.668737 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:34.668812 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:34.669264 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 23:06:34.669361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:34.670370 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.657542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40665 (local address 127.14.75.3:45493)
0810 23:06:34.658007 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:34.658052 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:34.659350 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:34.659882 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:34.659889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:34.660261 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:34.660743 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.660757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.661705 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.661708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.662919 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:34.662929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.663449 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.663453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:34.663652 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.664423 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:34.664444 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:34.666045 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:34.667836 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.667841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.667853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.668101 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.668342 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:34.668344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:34.668345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:34.668476 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:34.668821 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:34.668826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:34.669060 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 23:06:34.669217 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 23:06:34.669547 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260810 23:06:34.671499 18906 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40665
I20260810 23:06:34.671772 18906 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:34.672312 18906 heartbeater.cc:511] Master 127.14.75.62:40665 requested a full tablet report, sending...
I20260810 23:06:34.673395 18423 ts_manager.cc:194] Registered new tserver with Master: 912b6f14785944398be824a65a92ac4b (127.14.75.3:44859)
I20260810 23:06:34.674086 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:45493
I20260810 23:06:34.676168 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:34.702847 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.693249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.14.75.1:36633)
0810 23:06:34.693622 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:34.693647 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:34.693671 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:06:34.693722 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:34.693727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:34.693913 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:34.694031 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:34.694687 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.695466 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.696340 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.696703 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.696834 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.699322 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:34.699332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:34.699346 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:34.699372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:34.701162 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.701592 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.701597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.701598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.701681 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.701888 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.701890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.701891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.702139 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:34.702353 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:34.702469 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:06:34.702636 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":155,"threads_started":1}
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:34.713928 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:34.705852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.14.75.62:40665)
0810 23:06:34.705993 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:34.706000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:34.706037 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:34.706186 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:34.706190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:34.706238 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:34.706327 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:34.706917 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.707433 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.708200 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:34.708418 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:34.708521 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:34.710921 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:34.710924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:34.710926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:34.710951 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:34.712260 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.712632 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.712635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.712636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.712707 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:34.712927 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:34.712929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:34.712930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:34.713169 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:34.713277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:34.713614 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:06:34.713768 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:06:34.714381 18423 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50226
I20260810 23:06:34.715112 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18489
I20260810 23:06:34.721902 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18628
I20260810 23:06:34.729475 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18770
I20260810 23:06:34.736372 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18390
2026-08-10T23:06:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2198 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18918](info): set up 2 sockets
Aug 10 23:06:34 dist-test-slave-srgv krb5kdc[18918](info): commencing operation
krb5kdc: starting...
W20260810 23:06:35.782774 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:35 dist-test-slave-srgv krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:35.912376 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45671
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:36.027362 18934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:36.027737 18934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:36.027827 18934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:36.032115 18934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:36.032234 18934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:36.032270 18934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:36.032307 18934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:36.032344 18934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:36.037949 18934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:45671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:36.039122 18934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:36.040220 18934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:36.048897 18942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.048943 18934 server_base.cc:1034] running on GCE node
W20260810 23:06:36.048892 18939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:36.048905 18940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.049861 18934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:36.051013 18934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:36.052224 18934 hybrid_clock.cc:648] HybridClock initialized: now 1786403196052211 us; error 27 us; skew 500 ppm
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:36.060323 18934 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:36.061196 18934 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:36.061296 18934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:36.061582 18934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:36.063759 18934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "14057ec43b76417197b0a92e835a6adb"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "5b34a1c26ff9f893a78dc367b69b16b7"
server_key_iv: "8ae31dfa2dc4d366097837a299fcd090"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.064380 18934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "14057ec43b76417197b0a92e835a6adb"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "5b34a1c26ff9f893a78dc367b69b16b7"
server_key_iv: "8ae31dfa2dc4d366097837a299fcd090"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.068209 18934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:06:36.070719 18948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.071947 18934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:36.072089 18934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "14057ec43b76417197b0a92e835a6adb"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "5b34a1c26ff9f893a78dc367b69b16b7"
server_key_iv: "8ae31dfa2dc4d366097837a299fcd090"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.072206 18934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:36.088039 18934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:36.091068 18934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:36.091297 18934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:36.098567 18934 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:45671
I20260810 23:06:36.098867 19001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:45671 every 8 connection(s)
I20260810 23:06:36.099280 18934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:36.101977 19002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:36.107512 19002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: Bootstrap starting.
I20260810 23:06:36.109547 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 18934
I20260810 23:06:36.109691 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:36.109980 14636 external_mini_cluster.cc:1428] Setting key 711e8be845d3d2b98da7e94d9cb13c9d
I20260810 23:06:36.110267 19002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:36.111217 19002 log.cc:824] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:36.113294 19002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: No bootstrap required, opened a new log
I20260810 23:06:36.117524 19002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } }
I20260810 23:06:36.117820 19002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:36.117894 19002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14057ec43b76417197b0a92e835a6adb, State: Initialized, Role: FOLLOWER
I20260810 23:06:36.118422 19002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } }
I20260810 23:06:36.118609 19002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:36.118680 19002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:36.118798 19002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:36.119890 19002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } }
I20260810 23:06:36.120230 19002 leader_election.cc:302] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14057ec43b76417197b0a92e835a6adb; no voters:
I20260810 23:06:36.120550 19002 leader_election.cc:288] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:36.120745 19007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:36.121304 19007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [term 1 LEADER]: Becoming Leader. State: Replica: 14057ec43b76417197b0a92e835a6adb, State: Running, Role: LEADER
I20260810 23:06:36.121718 19007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } }
I20260810 23:06:36.121820 19002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:36.123510 19009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14057ec43b76417197b0a92e835a6adb. Latest consensus state: current_term: 1 leader_uuid: "14057ec43b76417197b0a92e835a6adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } } }
I20260810 23:06:36.123656 19009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:36.123979 19008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14057ec43b76417197b0a92e835a6adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14057ec43b76417197b0a92e835a6adb" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45671 } } }
I20260810 23:06:36.124107 19008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:36.126098 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.111172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.14.75.62:45671)
0810 23:06:36.111635 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.111651 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.111686 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:06:36.112358 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.112374 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.112681 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.112976 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:36.113929 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.114892 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.116274 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.116984 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.117125 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.120004 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:36.120011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:36.120048 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:36.120080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:36.122283 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.122834 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.122841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.122847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.122991 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.123320 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.123323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.123325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.123843 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:36.124048 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.124835 (+ 787us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.125123 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":179,"threads_started":1}
W20260810 23:06:36.127316 19023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:36.127393 19023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:36.127449 19021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:36.128245 19021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:36.133788 19021 catalog_manager.cc:1347] Generated new cluster ID: 45a57420c5ac44a583ce564f697f681a
I20260810 23:06:36.133859 19021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:36.148069 19021 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:36.149398 19021 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:36.159176 19021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: Generated new TSK 0
I20260810 23:06:36.160094 19021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:36.192785 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:36.307317 19029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:36.307654 19029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:36.307744 19029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:36.311980 19029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:36.312083 19029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:36.312179 19029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:36.317864 19029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:36.319027 19029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:36.320099 19029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:36.330017 19036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:36.330017 19035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.330313 19029 server_base.cc:1034] running on GCE node
W20260810 23:06:36.330021 19038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.330813 19029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:36.331488 19029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:36.332683 19029 hybrid_clock.cc:648] HybridClock initialized: now 1786403196332663 us; error 32 us; skew 500 ppm
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:36.335777 19029 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:36.336561 19029 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:36.336629 19029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:36.336812 19029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:36.338929 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "80528b5a42f545c582655ef706165878"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "97044a155e28ff3a8c2c9328f87657d1"
server_key_iv: "aebfbe9e1ee98efebf1b5ad34de64f2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.339461 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "80528b5a42f545c582655ef706165878"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "97044a155e28ff3a8c2c9328f87657d1"
server_key_iv: "aebfbe9e1ee98efebf1b5ad34de64f2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.343438 19029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:06:36.346369 19044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.347663 19029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:36.347796 19029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "80528b5a42f545c582655ef706165878"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "97044a155e28ff3a8c2c9328f87657d1"
server_key_iv: "aebfbe9e1ee98efebf1b5ad34de64f2d"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.347923 19029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:36.369356 19029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:36.372852 19029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:36.373075 19029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:36.373778 19029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:36.374660 19029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:36.374730 19029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.374796 19029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:36.374843 19029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.383780 19029 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:44407
I20260810 23:06:36.383801 19161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:44407 every 8 connection(s)
I20260810 23:06:36.384430 19029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:36.390448 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19029
I20260810 23:06:36.390585 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:36.390890 14636 external_mini_cluster.cc:1428] Setting key bd2e603f7402d510a606b902d25c7dfb
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:36.399353 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.386274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:33621 (local address 127.14.75.62:45671)
0810 23:06:36.386423 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.386427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.387252 (+ 825us) server_negotiation.cc:407] Connection header received
0810 23:06:36.388579 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.388583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.388643 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.388757 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:36.390053 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.390917 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.392482 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.392811 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.392928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.395526 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:36.395529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:36.395531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:36.395555 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:36.397084 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.397733 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.397737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.397739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.397823 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.398187 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.398190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.398192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.398391 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:36.398504 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.399037 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.399176 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:06:36.400209 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.386637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45671 (local address 127.14.75.1:33621)
0810 23:06:36.387102 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.387145 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.388359 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.388934 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.388942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.389331 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:36.389898 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.389912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.391066 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.391070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.392347 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.392360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.392970 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.392973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.393116 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.393807 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:36.393825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:36.395341 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:36.397227 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.397235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.397252 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.397623 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.397922 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.397927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.397929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.398080 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.398504 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:36.398512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:36.398782 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.398978 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.399333 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:36.401657 19162 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45671
I20260810 23:06:36.401898 19162 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:36.402330 19162 heartbeater.cc:511] Master 127.14.75.62:45671 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:36.403928 18965 ts_manager.cc:194] Registered new tserver with Master: 80528b5a42f545c582655ef706165878 (127.14.75.1:44407)
I20260810 23:06:36.405067 18965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:33621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:36.420543 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:36.536422 19167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:36.536684 19167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:36.536731 19167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:36.541121 19167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:36.541215 19167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:36.541281 19167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:36.546830 19167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:36.547904 19167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:36.548894 19167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:36.558810 19172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:36.558858 19173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.558908 19167 server_base.cc:1034] running on GCE node
W20260810 23:06:36.558810 19175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.559573 19167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:36.560290 19167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:36.561482 19167 hybrid_clock.cc:648] HybridClock initialized: now 1786403196561468 us; error 27 us; skew 500 ppm
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:36.564415 19167 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:36.565177 19167 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:36.565277 19167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:36.565608 19167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:36.567850 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "5344d07baead4d25b40b2cc546ffe214"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "77b4e4f0a2c4adb275461d009ef3cab7"
server_key_iv: "a189c26b2e596f27018ce03f05b33f15"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.568476 19167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "5344d07baead4d25b40b2cc546ffe214"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "77b4e4f0a2c4adb275461d009ef3cab7"
server_key_iv: "a189c26b2e596f27018ce03f05b33f15"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.572729 19167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:36.575469 19181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.576992 19167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:06:36.577132 19167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "5344d07baead4d25b40b2cc546ffe214"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "77b4e4f0a2c4adb275461d009ef3cab7"
server_key_iv: "a189c26b2e596f27018ce03f05b33f15"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.577265 19167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:36.587538 19167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:36.590984 19167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:36.591199 19167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:36.591791 19167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:36.592629 19167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:36.592706 19167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.592775 19167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:36.592828 19167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.601974 19167 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36759
I20260810 23:06:36.602039 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36759 every 8 connection(s)
I20260810 23:06:36.602639 19167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:36.607985 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19167
I20260810 23:06:36.608131 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:36.608443 14636 external_mini_cluster.cc:1428] Setting key 5d9eceda88ee87985f6c372ab4d9e09d
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:36.617527 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.604461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:41909 (local address 127.14.75.62:45671)
0810 23:06:36.604607 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.604620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.605439 (+ 819us) server_negotiation.cc:407] Connection header received
0810 23:06:36.606737 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.606741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.606803 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.606916 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:36.608126 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.608755 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.610190 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.610410 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.610521 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.613440 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:36.613447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:36.613453 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:36.613488 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:36.615268 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.615793 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.615799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.615801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.615903 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.616244 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.616249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.616252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.616538 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:36.616671 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.617159 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.617300 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:06:36.618362 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.604812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45671 (local address 127.14.75.2:41909)
0810 23:06:36.605277 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.605321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.606527 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.607088 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.607094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.607465 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:36.607974 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.607988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.608885 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.608888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.610057 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.610067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.610564 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.610568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.610748 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.611428 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:36.611445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:36.613228 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:36.615392 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.615397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.615411 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.615684 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.616005 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.616007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.616008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.616146 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.616652 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:36.616658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:36.616906 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.617067 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.617352 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:36.619794 19302 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45671
I20260810 23:06:36.620013 19302 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:36.620484 19302 heartbeater.cc:511] Master 127.14.75.62:45671 requested a full tablet report, sending...
I20260810 23:06:36.621534 18965 ts_manager.cc:194] Registered new tserver with Master: 5344d07baead4d25b40b2cc546ffe214 (127.14.75.2:36759)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:06:36.622289 18965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:41909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:36.638564 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:36.751263 19308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:36.751680 19308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:36.751763 19308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:36.755986 19308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:36.756054 19308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:36.756170 19308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:36.761610 19308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:36.762774 19308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:36.763813 19308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:36.773453 19315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.773491 19308 server_base.cc:1034] running on GCE node
W20260810 23:06:36.773438 19316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:36.773442 19318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:36.774183 19308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:36.774838 19308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:36.776024 19308 hybrid_clock.cc:648] HybridClock initialized: now 1786403196775994 us; error 44 us; skew 500 ppm
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:36.778931 19308 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:36.779736 19308 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:36.779832 19308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:36.780084 19308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:36.782306 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "3657b3a7b29e455b9d95d7e3271d7e77"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "f7f1351d957770fbf40f20382bb841a1"
server_key_iv: "b08fd460ff664f5d2b968274296010f6"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.782981 19308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "3657b3a7b29e455b9d95d7e3271d7e77"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "f7f1351d957770fbf40f20382bb841a1"
server_key_iv: "b08fd460ff664f5d2b968274296010f6"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.787029 19308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:36.789614 19324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.791364 19308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:36.791507 19308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "3657b3a7b29e455b9d95d7e3271d7e77"
format_stamp: "Formatted at 2026-08-10 23:06:36 on dist-test-slave-srgv"
server_key: "f7f1351d957770fbf40f20382bb841a1"
server_key_iv: "b08fd460ff664f5d2b968274296010f6"
server_key_version: "encryptionkey@0"
I20260810 23:06:36.791652 19308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:36.815428 19308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:36.819155 19308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:36.819371 19308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:36.819978 19308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:36.820837 19308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:36.820910 19308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.820982 19308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:36.821038 19308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:36.830952 19308 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:38839
I20260810 23:06:36.830971 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:38839 every 8 connection(s)
I20260810 23:06:36.831599 19308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:36.835893 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19308
I20260810 23:06:36.836050 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:36.836339 14636 external_mini_cluster.cc:1428] Setting key dddb1f37bf5d5ad1de250a1201926b8b
Aug 10 23:06:36 dist-test-slave-srgv krb5kdc[18918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403196, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:36.845659 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.833354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40935 (local address 127.14.75.62:45671)
0810 23:06:36.833564 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.833568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.834415 (+ 847us) server_negotiation.cc:407] Connection header received
0810 23:06:36.835741 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.835745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.835803 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.835886 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:36.837240 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.837867 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.839018 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.839256 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.839375 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.842219 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:36.842223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:36.842226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:36.842259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:36.843519 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.844050 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.844056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.844057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.844144 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.844499 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.844503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.844505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.844707 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:36.844820 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.845334 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.845483 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 23:06:36.846599 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.833741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45671 (local address 127.14.75.3:40935)
0810 23:06:36.834242 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.834287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.835519 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.836103 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.836135 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.836583 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:36.837079 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.837094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.838008 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.838013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.838881 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.838891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.839424 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.839436 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.839604 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.840454 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:36.840471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:36.842027 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:36.843645 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.843651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.843666 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.843930 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.844242 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:36.844245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:36.844248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:36.844396 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:36.844819 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:36.844828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:36.845085 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.845293 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.845667 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
I20260810 23:06:36.847805 19444 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45671
I20260810 23:06:36.848069 19444 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:36.848591 19444 heartbeater.cc:511] Master 127.14.75.62:45671 requested a full tablet report, sending...
I20260810 23:06:36.849785 18965 ts_manager.cc:194] Registered new tserver with Master: 3657b3a7b29e455b9d95d7e3271d7e77 (127.14.75.3:38839)
I20260810 23:06:36.850359 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:36.850555 18965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:40935
I20260810 23:06:36.859822 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.852651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.14.75.62:45671)
0810 23:06:36.852802 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.852807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.852996 (+ 189us) server_negotiation.cc:407] Connection header received
0810 23:06:36.853084 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.853086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.853124 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.853192 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:36.853964 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.854467 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.855243 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.855446 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.855522 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.856832 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:36.856835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:36.856837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:36.856859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:36.858282 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.858678 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.858680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.858681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.858756 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:36.858984 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:36.858986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:36.858987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:36.859193 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:36.859293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.859523 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.859615 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:36.862854 18965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:36.865954 18965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:36.880913 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.874293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:38839 (local address 127.0.0.1:45702)
0810 23:06:36.874787 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.874838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.874973 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.876394 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.876397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.876416 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:36.876533 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.876540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.877648 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.877651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.879092 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.879103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.880130 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.880396 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880467 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.880582 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.880742 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":88,"thread_start_us":88,"threads_started":1}
I20260810 23:06:36.881002 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.875378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.14.75.2:36759)
0810 23:06:36.875848 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.875861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.875883 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:06:36.875957 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.875962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.876140 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.876320 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:36.876784 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.877871 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.879111 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.879978 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880099 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880261 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.880726 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.880829 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":159,"threads_started":1}
I20260810 23:06:36.881217 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.874470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.14.75.3:38839)
0810 23:06:36.875997 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.876009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.876028 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:36.876092 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.876096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.876257 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.876414 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:36.876670 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.877519 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.879232 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.879984 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880261 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.880633 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.880741 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434,"thread_start_us":140,"threads_started":1}
I20260810 23:06:36.881328 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.875096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.14.75.1:44407)
0810 23:06:36.875546 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.875562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.875589 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:06:36.875672 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.875676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.875853 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.876020 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:36.876395 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.877847 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.879467 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.880018 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880683 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880874 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.880924 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.881017 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":105,"threads_started":1}
I20260810 23:06:36.881980 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.874476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:44407 (local address 127.0.0.1:50364)
0810 23:06:36.875351 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.875364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.875451 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.876017 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.876023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.876046 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:36.876226 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.876235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.878017 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.878021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.879346 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.879353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.880188 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.880320 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880443 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.880553 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.881870 (+ 1317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":1378,"thread_start_us":105,"threads_started":1}
I20260810 23:06:36.882315 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.875026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36759 (local address 127.0.0.1:33850)
0810 23:06:36.875661 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.875675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.875759 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.876511 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.876514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.876535 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:36.876652 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.876659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.878017 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.878021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.878981 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.878991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.880334 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.880337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.880544 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.880569 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.880631 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.882186 (+ 1555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":1557,"thread_start_us":104,"threads_started":1}
I20260810 23:06:36.884565 19374 tablet_service.cc:1467] Processing CreateTablet for tablet afd4883781714e4db1688f9412fbebd2 (DEFAULT_TABLE table=test-table [id=78950305225d4b3fa9976c16879cb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:36.884550 19233 tablet_service.cc:1467] Processing CreateTablet for tablet afd4883781714e4db1688f9412fbebd2 (DEFAULT_TABLE table=test-table [id=78950305225d4b3fa9976c16879cb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:36.884585 19095 tablet_service.cc:1467] Processing CreateTablet for tablet afd4883781714e4db1688f9412fbebd2 (DEFAULT_TABLE table=test-table [id=78950305225d4b3fa9976c16879cb8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:36.885739 19095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd4883781714e4db1688f9412fbebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:36.885732 19233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd4883781714e4db1688f9412fbebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:36.885736 19374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afd4883781714e4db1688f9412fbebd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:36.892208 19461 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Bootstrap starting.
I20260810 23:06:36.893746 19463 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: Bootstrap starting.
I20260810 23:06:36.894528 19461 tablet_bootstrap.cc:654] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:36.895743 19463 tablet_bootstrap.cc:654] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:36.895815 19461 log.cc:824] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:36.896613 19462 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Bootstrap starting.
I20260810 23:06:36.897051 19463 log.cc:824] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:36.898280 19461 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: No bootstrap required, opened a new log
I20260810 23:06:36.898483 19461 ts_tablet_manager.cc:1397] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:06:36.898758 19462 tablet_bootstrap.cc:654] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:36.899137 19463 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: No bootstrap required, opened a new log
I20260810 23:06:36.899334 19463 ts_tablet_manager.cc:1397] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260810 23:06:36.899870 19462 log.cc:824] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:36.901587 19462 tablet_bootstrap.cc:492] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: No bootstrap required, opened a new log
I20260810 23:06:36.901786 19462 ts_tablet_manager.cc:1397] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:06:36.902607 19461 raft_consensus.cc:348] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.902956 19461 raft_consensus.cc:374] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:36.903038 19461 raft_consensus.cc:729] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5344d07baead4d25b40b2cc546ffe214, State: Initialized, Role: FOLLOWER
I20260810 23:06:36.903318 19463 raft_consensus.cc:348] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.903579 19463 raft_consensus.cc:374] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:36.903568 19461 consensus_queue.cc:260] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.903679 19463 raft_consensus.cc:729] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80528b5a42f545c582655ef706165878, State: Initialized, Role: FOLLOWER
I20260810 23:06:36.904381 19302 heartbeater.cc:503] Master 127.14.75.62:45671 was elected leader, sending a full tablet report...
I20260810 23:06:36.904316 19463 consensus_queue.cc:260] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.904775 19461 ts_tablet_manager.cc:1428] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:06:36.904968 19162 heartbeater.cc:503] Master 127.14.75.62:45671 was elected leader, sending a full tablet report...
I20260810 23:06:36.906232 19463 ts_tablet_manager.cc:1428] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260810 23:06:36.906656 19462 raft_consensus.cc:348] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.906939 19462 raft_consensus.cc:374] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:36.906997 19462 raft_consensus.cc:729] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3657b3a7b29e455b9d95d7e3271d7e77, State: Initialized, Role: FOLLOWER
I20260810 23:06:36.907465 19462 consensus_queue.cc:260] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.908257 19444 heartbeater.cc:503] Master 127.14.75.62:45671 was elected leader, sending a full tablet report...
I20260810 23:06:36.908454 19462 ts_tablet_manager.cc:1428] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:06:36.979868 19467 raft_consensus.cc:482] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:36.980147 19467 raft_consensus.cc:504] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.981462 19467 leader_election.cc:288] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3657b3a7b29e455b9d95d7e3271d7e77 (127.14.75.3:38839), 80528b5a42f545c582655ef706165878 (127.14.75.1:44407)
I20260810 23:06:36.986793 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.981923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:50373 (local address 127.14.75.1:44407)
0810 23:06:36.982064 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.982070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.982260 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:06:36.982324 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.982330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.982387 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.982463 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:36.982717 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.983521 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.985458 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.986069 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.986184 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.986251 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.986575 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.986651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:36.986793 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.981829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:47483 (local address 127.14.75.3:38839)
0810 23:06:36.981967 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:36.981973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:36.981990 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:36.982186 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:36.982189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:36.982286 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:36.982377 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:36.982717 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.983521 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.985462 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.986068 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.986183 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.986256 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:36.986575 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:06:36.986651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:36.986835 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.981703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:38839 (local address 127.14.75.2:47483)
0810 23:06:36.981911 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.981927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.982031 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.982402 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.982404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.982418 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:36.982581 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.982589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.983647 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.983651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.985295 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.985309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.986205 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.986209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.986398 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.986464 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.986544 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.986658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":235}
I20260810 23:06:36.986841 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:36.981759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:44407 (local address 127.14.75.2:50373)
0810 23:06:36.982158 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:36.982178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:36.982254 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:36.982490 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:36.982493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:36.982504 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:36.982601 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.982607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.983653 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.983656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.985298 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:36.985309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:36.986205 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:36.986209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:36.986414 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:36.986469 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 23:06:36.986544 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:06:36.986667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":327,"thread_start_us":149,"threads_started":1}
I20260810 23:06:36.987668 19116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd4883781714e4db1688f9412fbebd2" candidate_uuid: "5344d07baead4d25b40b2cc546ffe214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80528b5a42f545c582655ef706165878" is_pre_election: true
I20260810 23:06:36.987668 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd4883781714e4db1688f9412fbebd2" candidate_uuid: "5344d07baead4d25b40b2cc546ffe214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" is_pre_election: true
I20260810 23:06:36.988027 19116 raft_consensus.cc:2454] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5344d07baead4d25b40b2cc546ffe214 in term 0.
I20260810 23:06:36.988026 19395 raft_consensus.cc:2454] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5344d07baead4d25b40b2cc546ffe214 in term 0.
I20260810 23:06:36.988606 19185 leader_election.cc:302] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3657b3a7b29e455b9d95d7e3271d7e77, 5344d07baead4d25b40b2cc546ffe214, 80528b5a42f545c582655ef706165878; no voters:
I20260810 23:06:36.988953 19467 raft_consensus.cc:2781] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:36.989064 19467 raft_consensus.cc:482] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:36.989148 19467 raft_consensus.cc:3037] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:36.990263 19467 raft_consensus.cc:504] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.990639 19467 leader_election.cc:288] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [CANDIDATE]: Term 1 election: Requested vote from peers 3657b3a7b29e455b9d95d7e3271d7e77 (127.14.75.3:38839), 80528b5a42f545c582655ef706165878 (127.14.75.1:44407)
I20260810 23:06:36.991050 19116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd4883781714e4db1688f9412fbebd2" candidate_uuid: "5344d07baead4d25b40b2cc546ffe214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80528b5a42f545c582655ef706165878"
I20260810 23:06:36.991086 19395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afd4883781714e4db1688f9412fbebd2" candidate_uuid: "5344d07baead4d25b40b2cc546ffe214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3657b3a7b29e455b9d95d7e3271d7e77"
I20260810 23:06:36.991199 19116 raft_consensus.cc:3037] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:36.991231 19395 raft_consensus.cc:3037] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:36.992290 19395 raft_consensus.cc:2454] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5344d07baead4d25b40b2cc546ffe214 in term 1.
I20260810 23:06:36.992302 19116 raft_consensus.cc:2454] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5344d07baead4d25b40b2cc546ffe214 in term 1.
I20260810 23:06:36.992653 19182 leader_election.cc:302] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3657b3a7b29e455b9d95d7e3271d7e77, 5344d07baead4d25b40b2cc546ffe214; no voters:
I20260810 23:06:36.992901 19467 raft_consensus.cc:2781] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:36.993140 19467 raft_consensus.cc:686] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 1 LEADER]: Becoming Leader. State: Replica: 5344d07baead4d25b40b2cc546ffe214, State: Running, Role: LEADER
I20260810 23:06:36.993531 19467 consensus_queue.cc:237] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } }
I20260810 23:06:36.996901 18963 catalog_manager.cc:5495] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5344d07baead4d25b40b2cc546ffe214 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "5344d07baead4d25b40b2cc546ffe214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5344d07baead4d25b40b2cc546ffe214" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36759 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:37.033260 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.029953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.14.75.2:36759)
0810 23:06:37.030088 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.030092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.030123 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:06:37.030208 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.030211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.030270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.030356 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:37.030595 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:37.031101 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:37.031931 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:37.032151 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:37.032228 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:37.032492 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:37.032905 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:37.033000 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.033056 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.033109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:37.039191 19116 raft_consensus.cc:1261] T afd4883781714e4db1688f9412fbebd2 P 80528b5a42f545c582655ef706165878 [term 1 FOLLOWER]: Refusing update from remote peer 5344d07baead4d25b40b2cc546ffe214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:37.039192 19395 raft_consensus.cc:1261] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 1 FOLLOWER]: Refusing update from remote peer 5344d07baead4d25b40b2cc546ffe214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:37.039945 19471 consensus_queue.cc:1035] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3657b3a7b29e455b9d95d7e3271d7e77" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 38839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:37.040159 19467 consensus_queue.cc:1035] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80528b5a42f545c582655ef706165878" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:37.048065 19476 mvcc.cc:205] Tried to move back new op lower bound from 7317107495064903680 to 7317107494888108032. Current Snapshot: MvccSnapshot[applied={T|T < 7317107495064903680}]
I20260810 23:06:37.057526 18964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:37.057729 18964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:37.060276 18964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14057ec43b76417197b0a92e835a6adb: Sending DeleteTablet for 3 replicas of tablet afd4883781714e4db1688f9412fbebd2
I20260810 23:06:37.061151 19095 tablet_service.cc:1514] Processing DeleteTablet for tablet afd4883781714e4db1688f9412fbebd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:37 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:50364
I20260810 23:06:37.061213 19374 tablet_service.cc:1514] Processing DeleteTablet for tablet afd4883781714e4db1688f9412fbebd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:37 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45702
I20260810 23:06:37.061316 19233 tablet_service.cc:1514] Processing DeleteTablet for tablet afd4883781714e4db1688f9412fbebd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:37 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:33850
I20260810 23:06:37.061781 19484 tablet_replica.cc:330] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: stopping tablet replica
I20260810 23:06:37.061951 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19029
I20260810 23:06:37.062127 19484 raft_consensus.cc:2229] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:37.062476 19484 raft_consensus.cc:2258] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:37.063046 19485 tablet_replica.cc:330] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: stopping tablet replica
I20260810 23:06:37.063347 19485 raft_consensus.cc:2229] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:37.063697 19485 raft_consensus.cc:2258] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:37.064663 19484 ts_tablet_manager.cc:1905] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:37.067284 19485 ts_tablet_manager.cc:1905] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:37.068238 19484 ts_tablet_manager.cc:1918] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:37.068434 19484 log.cc:1196] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/afd4883781714e4db1688f9412fbebd2
I20260810 23:06:37.068879 19484 ts_tablet_manager.cc:1939] T afd4883781714e4db1688f9412fbebd2 P 3657b3a7b29e455b9d95d7e3271d7e77: Deleting consensus metadata
I20260810 23:06:37.070017 19485 ts_tablet_manager.cc:1918] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:37.070348 19485 log.cc:1196] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/afd4883781714e4db1688f9412fbebd2
I20260810 23:06:37.070384 18949 catalog_manager.cc:4841] TS 3657b3a7b29e455b9d95d7e3271d7e77 (127.14.75.3:38839): tablet afd4883781714e4db1688f9412fbebd2 (table test-table [id=78950305225d4b3fa9976c16879cb8f7]) successfully deleted
I20260810 23:06:37.070781 19485 ts_tablet_manager.cc:1939] T afd4883781714e4db1688f9412fbebd2 P 5344d07baead4d25b40b2cc546ffe214: Deleting consensus metadata
W20260810 23:06:37.071695 18952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:44407 (error 108)
I20260810 23:06:37.071907 18949 catalog_manager.cc:4841] TS 5344d07baead4d25b40b2cc546ffe214 (127.14.75.2:36759): tablet afd4883781714e4db1688f9412fbebd2 (table test-table [id=78950305225d4b3fa9976c16879cb8f7]) successfully deleted
I20260810 23:06:37.072772 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19167
I20260810 23:06:37.073160 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.072793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:44407 (local address 127.0.0.1:50368)
0810 23:06:37.072962 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.073046 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:44407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:06:37.074769 18952 catalog_manager.cc:4584] TS 80528b5a42f545c582655ef706165878 (127.14.75.1:44407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afd4883781714e4db1688f9412fbebd2: Network error: Client connection negotiation failed: client connection to 127.14.75.1:44407: connect: Connection refused (error 111)
I20260810 23:06:37.078735 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19308
I20260810 23:06:37.085482 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 18934
2026-08-10T23:06:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2348 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:37Z Disabled control of system clock
I20260810 23:06:37.116976 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:06:37.227744 19490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:37.228035 19490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:37.228075 19490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:37.232120 19490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:37.232187 19490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:37.232201 19490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:37.232218 19490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:37.232234 19490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:37.237039 19490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:37.238284 19490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:37.239707 19490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:37.248652 19490 server_base.cc:1034] running on GCE node
W20260810 23:06:37.248608 19495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.248596 19498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.248596 19496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:37.249684 19490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:37.250806 19490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:37.252028 19490 hybrid_clock.cc:648] HybridClock initialized: now 1786403197252008 us; error 32 us; skew 500 ppm
I20260810 23:06:37.254168 19490 webserver.cc:460] Webserver started at http://127.14.75.62:39275/ using document root <none> and password file <none>
I20260810 23:06:37.254817 19490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:37.254911 19490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:37.255172 19490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:37.257335 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "fe0c965e87fbb5eabec86c9f1a50ec66"
server_key_iv: "6c27b8b9e86f5337cbf400576e100e49"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.257987 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "fe0c965e87fbb5eabec86c9f1a50ec66"
server_key_iv: "6c27b8b9e86f5337cbf400576e100e49"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.261848 19490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:06:37.264343 19504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.265666 19490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:37.265806 19490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "fe0c965e87fbb5eabec86c9f1a50ec66"
server_key_iv: "6c27b8b9e86f5337cbf400576e100e49"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.265925 19490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:37.270465 19490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:37.271083 19490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:37.271267 19490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:37.277988 19490 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:38101
I20260810 23:06:37.278000 19559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:38101 every 8 connection(s)
I20260810 23:06:37.278939 19490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:37.282152 19560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:37.282995 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19490
I20260810 23:06:37.283115 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:37.283406 14636 external_mini_cluster.cc:1428] Setting key d426bc74add19fc094e246b5307ac64c
I20260810 23:06:37.289075 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: Bootstrap starting.
I20260810 23:06:37.289861 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.284950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.14.75.62:38101)
0810 23:06:37.285665 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.285685 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.285730 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:06:37.286585 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.286607 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.287108 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.287470 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.288261 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.288267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.288281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.288293 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.288360 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.288382 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.288527 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.288595 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.288855 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260810 23:06:37.291643 19560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:37.292464 19560 log.cc:824] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:37.294560 19560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: No bootstrap required, opened a new log
I20260810 23:06:37.298184 19560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:06:37.298415 19560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:37.298497 19560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb2810744ce4861aa3bb9aaf06fa1d8, State: Initialized, Role: FOLLOWER
I20260810 23:06:37.298933 19560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:06:37.299075 19560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:37.299144 19560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:37.299268 19560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:37.300206 19560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:06:37.300547 19560 leader_election.cc:302] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fb2810744ce4861aa3bb9aaf06fa1d8; no voters:
I20260810 23:06:37.300853 19560 leader_election.cc:288] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:37.300994 19565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:37.301273 19565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb2810744ce4861aa3bb9aaf06fa1d8, State: Running, Role: LEADER
I20260810 23:06:37.301594 19565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:06:37.302129 19560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:37.303148 19566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:06:37.303418 19567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb2810744ce4861aa3bb9aaf06fa1d8. Latest consensus state: current_term: 1 leader_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb2810744ce4861aa3bb9aaf06fa1d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:06:37.303535 19566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:37.303630 19567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8 [sys.catalog]: This master's current role is: LEADER
W20260810 23:06:37.307008 19582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:37.307080 19582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:37.307165 19577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:37.308063 19577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:37.313076 19577 catalog_manager.cc:1347] Generated new cluster ID: b0a51e6b986242e5ba0fcce511547924
I20260810 23:06:37.313148 19577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:37.336985 19577 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:37.337370 19577 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:37.345376 19577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: Generated new TSK 0
I20260810 23:06:37.346203 19577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:37.358873 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--builtin_ntp_servers=127.14.75.20:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:06:37.472445 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:37.472776 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:37.472854 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:37.476975 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:37.477082 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:37.477181 19587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:37.482049 19587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:37.483254 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:37.484714 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:37.494472 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.494482 19596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.494482 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:37.494693 19587 server_base.cc:1034] running on GCE node
I20260810 23:06:37.495215 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:37.495841 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:37.497004 19587 hybrid_clock.cc:648] HybridClock initialized: now 1786403197496997 us; error 41 us; skew 500 ppm
I20260810 23:06:37.498979 19587 webserver.cc:460] Webserver started at http://127.14.75.1:33197/ using document root <none> and password file <none>
I20260810 23:06:37.499576 19587 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:37.499647 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:37.499907 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:37.502097 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "cb4ef206029146d299745ce884c317ad"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "55d0a7fa8a91a9fe87ff990b7dbf9cec"
server_key_iv: "fbcaa5bffb9a5246462a452123e76877"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.502738 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "cb4ef206029146d299745ce884c317ad"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "55d0a7fa8a91a9fe87ff990b7dbf9cec"
server_key_iv: "fbcaa5bffb9a5246462a452123e76877"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.507139 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:37.510457 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.511734 19587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 23:06:37.511885 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "cb4ef206029146d299745ce884c317ad"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "55d0a7fa8a91a9fe87ff990b7dbf9cec"
server_key_iv: "fbcaa5bffb9a5246462a452123e76877"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.512015 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:37.516852 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:37.517549 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:37.517748 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:37.518296 19587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:37.519220 19587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:37.519292 19587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.519362 19587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:37.519414 19587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.528671 19587 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:41115
I20260810 23:06:37.528693 19718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:41115 every 8 connection(s)
I20260810 23:06:37.529644 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:37.535743 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.531503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:39991 (local address 127.14.75.62:38101)
0810 23:06:37.531646 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.531650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.532754 (+ 1104us) server_negotiation.cc:407] Connection header received
0810 23:06:37.534027 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.534030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.534067 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.534145 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.535281 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.535284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.535286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.535294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.535310 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.535316 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.535386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.535541 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.535591 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:37.535784 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19587
I20260810 23:06:37.535889 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:37.536164 14636 external_mini_cluster.cc:1428] Setting key 7ffa8dd0a0bb83d4add5b3215795b6c6
I20260810 23:06:37.536638 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.531873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.1:39991)
0810 23:06:37.532525 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.532566 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.533837 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.534300 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.534306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.534389 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.535043 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.535045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.535090 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.535156 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.535386 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.535390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.535394 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.535513 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.535701 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260810 23:06:37.538009 19719 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:06:37.538272 19719 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:37.538357 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--builtin_ntp_servers=127.14.75.20:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:06:37.538754 19719 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:06:37.540268 19522 ts_manager.cc:194] Registered new tserver with Master: cb4ef206029146d299745ce884c317ad (127.14.75.1:41115)
W20260810 23:06:37.652829 19722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:37.653112 19722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:37.653154 19722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:37.657317 19722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:37.657382 19722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:37.657460 19722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:37.662279 19722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:37.663393 19722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:37.664865 19722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:37.674749 19728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:37.674882 19722 server_base.cc:1034] running on GCE node
W20260810 23:06:37.674765 19727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.674780 19730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:37.675565 19722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:37.676230 19722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:37.677491 19722 hybrid_clock.cc:648] HybridClock initialized: now 1786403197677439 us; error 49 us; skew 500 ppm
I20260810 23:06:37.679437 19722 webserver.cc:460] Webserver started at http://127.14.75.2:33235/ using document root <none> and password file <none>
I20260810 23:06:37.680044 19722 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:37.680141 19722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:37.680382 19722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:37.682600 19722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "378bb91f239b44df8878cbb629a46456"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "36714b00854a85cf8abf3f46c5c52abd"
server_key_iv: "5e44ae99c4562fd69f00eaa04ac2f7b6"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.683208 19722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "378bb91f239b44df8878cbb629a46456"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "36714b00854a85cf8abf3f46c5c52abd"
server_key_iv: "5e44ae99c4562fd69f00eaa04ac2f7b6"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.687104 19722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:37.689780 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.691067 19722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:06:37.691255 19722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "378bb91f239b44df8878cbb629a46456"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "36714b00854a85cf8abf3f46c5c52abd"
server_key_iv: "5e44ae99c4562fd69f00eaa04ac2f7b6"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.691392 19722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:37.696215 19722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:37.696870 19722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:37.697068 19722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:37.697715 19722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:37.698653 19722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:37.698724 19722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.698796 19722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:37.698851 19722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.708261 19722 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:38599
I20260810 23:06:37.708339 19851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:38599 every 8 connection(s)
I20260810 23:06:37.709300 19722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:37.715575 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19722
I20260810 23:06:37.715725 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:37.715727 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.711235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:35161 (local address 127.14.75.62:38101)
0810 23:06:37.711384 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.711387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.712413 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 23:06:37.713832 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.713836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.713884 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.713993 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.715151 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.715154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.715157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.715171 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.715208 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.715218 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.715339 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.715506 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.715566 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:06:37.716094 14636 external_mini_cluster.cc:1428] Setting key 1c5b612aaf60afe5a095156cefef0097
I20260810 23:06:37.716946 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.711592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.2:35161)
0810 23:06:37.712260 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.712300 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.713611 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.714161 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.714167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.714256 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.714883 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.714884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.714926 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.714994 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.715338 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.715348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.715353 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.715520 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.715766 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260810 23:06:37.718135 19852 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:06:37.718366 19852 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:37.718640 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--builtin_ntp_servers=127.14.75.20:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:06:37.718858 19852 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:06:37.719914 19520 ts_manager.cc:194] Registered new tserver with Master: 378bb91f239b44df8878cbb629a46456 (127.14.75.2:38599)
W20260810 23:06:37.833709 19856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:37.834010 19856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:37.834054 19856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:37.838274 19856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:37.838342 19856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:37.838404 19856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:37.843343 19856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:37.844507 19856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:37.846056 19856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:37.855789 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.855801 19861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:37.855808 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:37.856127 19856 server_base.cc:1034] running on GCE node
I20260810 23:06:37.856549 19856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:37.857249 19856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:37.858448 19856 hybrid_clock.cc:648] HybridClock initialized: now 1786403197858422 us; error 39 us; skew 500 ppm
I20260810 23:06:37.860348 19856 webserver.cc:460] Webserver started at http://127.14.75.3:34629/ using document root <none> and password file <none>
I20260810 23:06:37.860944 19856 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:37.861040 19856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:37.861264 19856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:37.863412 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "244b41932c654a6ba49cf42fcd987e6d"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "dda799fa0d87e65d787ae5e495f7082c"
server_key_iv: "5aa7a5f41f7c3415c72f50b115b03f5c"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.864007 19856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "244b41932c654a6ba49cf42fcd987e6d"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "dda799fa0d87e65d787ae5e495f7082c"
server_key_iv: "5aa7a5f41f7c3415c72f50b115b03f5c"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.867877 19856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:37.870695 19870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.871989 19856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:37.872114 19856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "244b41932c654a6ba49cf42fcd987e6d"
format_stamp: "Formatted at 2026-08-10 23:06:37 on dist-test-slave-srgv"
server_key: "dda799fa0d87e65d787ae5e495f7082c"
server_key_iv: "5aa7a5f41f7c3415c72f50b115b03f5c"
server_key_version: "encryptionkey@0"
I20260810 23:06:37.872241 19856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:37.876827 19856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:37.877501 19856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:37.877683 19856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:37.878232 19856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:37.879140 19856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:37.879187 19856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.879249 19856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:37.879281 19856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:37.888062 19856 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:39369
I20260810 23:06:37.888139 19992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:39369 every 8 connection(s)
I20260810 23:06:37.889058 19856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:37.895367 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.890952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:38807 (local address 127.14.75.62:38101)
0810 23:06:37.891126 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.891131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.892146 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 23:06:37.893551 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.893554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.893601 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.893695 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.894857 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.894860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.894863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.894876 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.894899 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.894911 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.895020 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.895172 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.895221 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:06:37.896086 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 19856
I20260810 23:06:37.896210 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:37.896442 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.891342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.3:38807)
0810 23:06:37.891993 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.892035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.893299 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.893878 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.893885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.893970 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.894598 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.894600 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.894639 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.894709 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.895013 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.895021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.895027 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.895227 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.895466 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260810 23:06:37.896519 14636 external_mini_cluster.cc:1428] Setting key f78db3d027adcc775250cfcebfdd2206
I20260810 23:06:37.897738 19993 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:06:37.897995 19993 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:37.898484 19993 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:06:37.899480 19520 ts_manager.cc:194] Registered new tserver with Master: 244b41932c654a6ba49cf42fcd987e6d (127.14.75.3:39369)
I20260810 23:06:37.910702 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:37.914208 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.912655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.14.75.62:38101)
0810 23:06:37.912794 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.912797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.912982 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:06:37.913072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.913074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.913108 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.913186 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.913888 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.913891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.913893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.913904 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.913923 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.913930 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.913999 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.914063 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.914094 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:06:37.917124 19520 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:37.920192 19520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:37.931964 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.928703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41115 (local address 127.0.0.1:53886)
0810 23:06:37.929984 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.929997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.930095 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.930733 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.930736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.930751 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.931152 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.931154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.931162 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.931179 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.931565 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.931578 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.931586 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.931714 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.931782 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":112,"threads_started":1}
I20260810 23:06:37.931973 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.929456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.14.75.1:41115)
0810 23:06:37.930333 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.930347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.930373 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:06:37.930430 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.930437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.930599 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.930747 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.931368 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.931373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.931385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.931395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.931437 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.931466 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.931583 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.931738 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.931814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":103,"threads_started":1}
I20260810 23:06:37.932309 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.928610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:39369 (local address 127.0.0.1:47308)
0810 23:06:37.929075 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.929132 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.929248 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.930186 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.930192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.930268 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.930986 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.930989 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.931010 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.931276 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.931628 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.931631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.931634 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.931741 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.931792 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260810 23:06:37.932592 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.929691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:38599 (local address 127.0.0.1:34676)
0810 23:06:37.930279 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:37.930291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:37.930357 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:37.931064 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:37.931068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:37.931083 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:37.931486 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:37.931488 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:37.931498 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:37.931533 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:37.932351 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:37.932355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:37.932359 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:06:37.932423 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:06:37.932458 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":82,"threads_started":1}
I20260810 23:06:37.932073 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.928770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.14.75.3:39369)
0810 23:06:37.929248 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.929261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.929288 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:06:37.929348 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.929354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.929583 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.929740 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.931395 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.931400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.931409 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.931421 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.931460 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.931492 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.931583 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.931862 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.931917 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":170,"threads_started":1}
I20260810 23:06:37.933211 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:37.929842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.14.75.2:38599)
0810 23:06:37.930625 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:37.930637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:37.930659 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:06:37.930713 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:37.930720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:37.930884 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:37.931329 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:37.932143 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:37.932151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:37.932163 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:37.932172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:37.932222 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:37.932244 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:37.932944 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:37.932989 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 23:06:37.933054 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":116,"threads_started":1}
I20260810 23:06:37.935312 19920 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 (DEFAULT_TABLE table=test-table [id=ebc5597bea714527990665c5c7865cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:37.935413 19650 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 (DEFAULT_TABLE table=test-table [id=ebc5597bea714527990665c5c7865cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:37.935828 19785 tablet_service.cc:1467] Processing CreateTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 (DEFAULT_TABLE table=test-table [id=ebc5597bea714527990665c5c7865cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:37.936462 19920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb15dbdd0d4d19ba5acf772f5c1793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:37.936487 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb15dbdd0d4d19ba5acf772f5c1793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:37.936704 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58fb15dbdd0d4d19ba5acf772f5c1793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:37.942570 20011 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Bootstrap starting.
I20260810 23:06:37.944095 20012 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Bootstrap starting.
I20260810 23:06:37.945079 20011 tablet_bootstrap.cc:654] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:37.946039 20011 log.cc:824] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:37.946396 20012 tablet_bootstrap.cc:654] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:37.946759 20013 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Bootstrap starting.
I20260810 23:06:37.947474 20012 log.cc:824] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:37.949106 20013 tablet_bootstrap.cc:654] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:37.949352 20011 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: No bootstrap required, opened a new log
I20260810 23:06:37.949594 20011 ts_tablet_manager.cc:1397] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:06:37.950166 20012 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: No bootstrap required, opened a new log
I20260810 23:06:37.950333 20012 ts_tablet_manager.cc:1397] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:06:37.951041 20013 log.cc:824] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:37.954340 20011 raft_consensus.cc:348] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.954550 20011 raft_consensus.cc:374] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:37.954581 20011 raft_consensus.cc:729] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244b41932c654a6ba49cf42fcd987e6d, State: Initialized, Role: FOLLOWER
I20260810 23:06:37.954670 20013 tablet_bootstrap.cc:492] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: No bootstrap required, opened a new log
I20260810 23:06:37.954854 20013 ts_tablet_manager.cc:1397] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260810 23:06:37.955000 20011 consensus_queue.cc:260] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.954972 20012 raft_consensus.cc:348] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.955235 20012 raft_consensus.cc:374] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:37.955286 20012 raft_consensus.cc:729] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4ef206029146d299745ce884c317ad, State: Initialized, Role: FOLLOWER
I20260810 23:06:37.955644 19993 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
I20260810 23:06:37.955749 20012 consensus_queue.cc:260] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.956018 20011 ts_tablet_manager.cc:1428] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:06:37.956676 20012 ts_tablet_manager.cc:1428] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:06:37.957064 19719 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
I20260810 23:06:37.959838 20013 raft_consensus.cc:348] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.960175 20013 raft_consensus.cc:374] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:37.960240 20013 raft_consensus.cc:729] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378bb91f239b44df8878cbb629a46456, State: Initialized, Role: FOLLOWER
I20260810 23:06:37.960685 20013 consensus_queue.cc:260] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:37.961401 19852 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
I20260810 23:06:37.961675 20013 ts_tablet_manager.cc:1428] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 23:06:38.030930 19720 tablet.cc:2369] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:38.140233 19994 tablet.cc:2369] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:38.144516 20020 raft_consensus.cc:482] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:38.144732 20020 raft_consensus.cc:504] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:38.145839 20020 leader_election.cc:288] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244b41932c654a6ba49cf42fcd987e6d (127.14.75.3:39369), cb4ef206029146d299745ce884c317ad (127.14.75.1:41115)
I20260810 23:06:38.147897 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.146262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:50295 (local address 127.14.75.1:41115)
0810 23:06:38.146390 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.146396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.146734 (+ 338us) server_negotiation.cc:407] Connection header received
0810 23:06:38.146784 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.146786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.146818 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.146927 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:38.147466 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:38.147469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:38.147472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:38.147484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:38.147502 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:38.147513 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:38.147605 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:38.147721 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:06:38.147764 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:06:38.147897 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.146252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:57357 (local address 127.14.75.3:39369)
0810 23:06:38.146383 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.146389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.146410 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:06:38.146497 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.146500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.146535 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.146600 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:38.147430 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:38.147436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:38.147440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:38.147454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:38.147477 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:38.147487 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:38.147593 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:38.147722 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 23:06:38.147764 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:06:38.147899 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.146139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:39369 (local address 127.14.75.2:57357)
0810 23:06:38.146266 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:38.146283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:38.146375 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:38.146671 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:38.146676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:38.146692 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:38.147206 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:38.147208 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:38.147238 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:38.147272 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:38.147597 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:38.147600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:38.147603 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:06:38.147692 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:06:38.147743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 23:06:38.147919 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.146154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41115 (local address 127.14.75.2:50295)
0810 23:06:38.146619 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:38.146632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:38.146712 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:38.146952 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:38.146955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:38.146966 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:38.147305 (+ 339us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:06:38.147309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:38.147321 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:06:38.147345 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:38.147597 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:38.147600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:38.147603 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:06:38.147692 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:06:38.147752 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":133,"thread_start_us":115,"threads_started":1}
I20260810 23:06:38.148635 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb15dbdd0d4d19ba5acf772f5c1793" candidate_uuid: "378bb91f239b44df8878cbb629a46456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4ef206029146d299745ce884c317ad" is_pre_election: true
I20260810 23:06:38.148656 19942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb15dbdd0d4d19ba5acf772f5c1793" candidate_uuid: "378bb91f239b44df8878cbb629a46456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244b41932c654a6ba49cf42fcd987e6d" is_pre_election: true
I20260810 23:06:38.148957 19670 raft_consensus.cc:2454] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378bb91f239b44df8878cbb629a46456 in term 0.
I20260810 23:06:38.148957 19942 raft_consensus.cc:2454] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378bb91f239b44df8878cbb629a46456 in term 0.
I20260810 23:06:38.149482 19738 leader_election.cc:302] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378bb91f239b44df8878cbb629a46456, cb4ef206029146d299745ce884c317ad; no voters:
I20260810 23:06:38.149796 20020 raft_consensus.cc:2781] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:38.149883 20020 raft_consensus.cc:482] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:38.149936 20020 raft_consensus.cc:3037] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:38.151058 20020 raft_consensus.cc:504] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:38.151454 20020 leader_election.cc:288] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [CANDIDATE]: Term 1 election: Requested vote from peers 244b41932c654a6ba49cf42fcd987e6d (127.14.75.3:39369), cb4ef206029146d299745ce884c317ad (127.14.75.1:41115)
I20260810 23:06:38.151777 19942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb15dbdd0d4d19ba5acf772f5c1793" candidate_uuid: "378bb91f239b44df8878cbb629a46456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244b41932c654a6ba49cf42fcd987e6d"
I20260810 23:06:38.151921 19942 raft_consensus.cc:3037] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:38.151932 19670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58fb15dbdd0d4d19ba5acf772f5c1793" candidate_uuid: "378bb91f239b44df8878cbb629a46456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4ef206029146d299745ce884c317ad"
I20260810 23:06:38.152073 19670 raft_consensus.cc:3037] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:38.152992 19942 raft_consensus.cc:2454] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378bb91f239b44df8878cbb629a46456 in term 1.
I20260810 23:06:38.153180 19670 raft_consensus.cc:2454] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378bb91f239b44df8878cbb629a46456 in term 1.
I20260810 23:06:38.153304 19739 leader_election.cc:302] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244b41932c654a6ba49cf42fcd987e6d, 378bb91f239b44df8878cbb629a46456; no voters:
I20260810 23:06:38.153504 20020 raft_consensus.cc:2781] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:38.153743 20020 raft_consensus.cc:686] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 1 LEADER]: Becoming Leader. State: Replica: 378bb91f239b44df8878cbb629a46456, State: Running, Role: LEADER
I20260810 23:06:38.154064 20020 consensus_queue.cc:237] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } }
I20260810 23:06:38.157366 19520 catalog_manager.cc:5495] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 378bb91f239b44df8878cbb629a46456 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "378bb91f239b44df8878cbb629a46456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "378bb91f239b44df8878cbb629a46456" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 38599 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:38.167436 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.165988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.14.75.2:38599)
0810 23:06:38.166110 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.166114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.166156 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:06:38.166295 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.166297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.166352 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.166421 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:38.167126 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:38.167128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:38.167131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:38.167140 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:38.167155 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:38.167162 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:38.167230 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:38.167300 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:06:38.167333 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:06:38.173224 19942 raft_consensus.cc:1261] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 1 FOLLOWER]: Refusing update from remote peer 378bb91f239b44df8878cbb629a46456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:38.173230 19670 raft_consensus.cc:1261] T 58fb15dbdd0d4d19ba5acf772f5c1793 P cb4ef206029146d299745ce884c317ad [term 1 FOLLOWER]: Refusing update from remote peer 378bb91f239b44df8878cbb629a46456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:38.174014 20020 consensus_queue.cc:1035] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244b41932c654a6ba49cf42fcd987e6d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:38.174275 20022 consensus_queue.cc:1035] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb4ef206029146d299745ce884c317ad" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:38.183413 20026 mvcc.cc:205] Tried to move back new op lower bound from 7317107499710509056 to 7317107499641716736. Current Snapshot: MvccSnapshot[applied={T|T < 7317107499710509056}]
I20260810 23:06:38.192358 19520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:38.192507 19520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:38.195127 19520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fb2810744ce4861aa3bb9aaf06fa1d8: Sending DeleteTablet for 3 replicas of tablet 58fb15dbdd0d4d19ba5acf772f5c1793
I20260810 23:06:38.195977 19920 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:38 UTC) from {username='slave'} at 127.0.0.1:47308
I20260810 23:06:38.196494 20036 tablet_replica.cc:330] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: stopping tablet replica
I20260810 23:06:38.196506 19785 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:38 UTC) from {username='slave'} at 127.0.0.1:34676
I20260810 23:06:38.196832 20036 raft_consensus.cc:2229] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:38.196954 20037 tablet_replica.cc:330] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: stopping tablet replica
I20260810 23:06:38.197149 20036 raft_consensus.cc:2258] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:38.197216 20037 raft_consensus.cc:2229] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:38.197607 19650 tablet_service.cc:1514] Processing DeleteTablet for tablet 58fb15dbdd0d4d19ba5acf772f5c1793 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:38 UTC) from {username='slave'} at 127.0.0.1:53886
I20260810 23:06:38.197686 20037 raft_consensus.cc:2258] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:38.198541 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19587
I20260810 23:06:38.198869 20037 ts_tablet_manager.cc:1905] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:38.199486 20036 ts_tablet_manager.cc:1905] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:38.201922 20037 ts_tablet_manager.cc:1918] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:38.202008 20037 log.cc:1196] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/58fb15dbdd0d4d19ba5acf772f5c1793
I20260810 23:06:38.202395 20037 ts_tablet_manager.cc:1939] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 378bb91f239b44df8878cbb629a46456: Deleting consensus metadata
I20260810 23:06:38.202657 20036 ts_tablet_manager.cc:1918] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:38.202754 20036 log.cc:1196] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/58fb15dbdd0d4d19ba5acf772f5c1793
I20260810 23:06:38.203094 20036 ts_tablet_manager.cc:1939] T 58fb15dbdd0d4d19ba5acf772f5c1793 P 244b41932c654a6ba49cf42fcd987e6d: Deleting consensus metadata
I20260810 23:06:38.203900 19508 catalog_manager.cc:4841] TS 378bb91f239b44df8878cbb629a46456 (127.14.75.2:38599): tablet 58fb15dbdd0d4d19ba5acf772f5c1793 (table test-table [id=ebc5597bea714527990665c5c7865cbc]) successfully deleted
I20260810 23:06:38.204869 19507 catalog_manager.cc:4841] TS 244b41932c654a6ba49cf42fcd987e6d (127.14.75.3:39369): tablet 58fb15dbdd0d4d19ba5acf772f5c1793 (table test-table [id=ebc5597bea714527990665c5c7865cbc]) successfully deleted
W20260810 23:06:38.209992 19506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:41115 (error 108)
I20260810 23:06:38.210242 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19722
I20260810 23:06:38.211935 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.211496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41115 (local address 127.0.0.1:53898)
0810 23:06:38.211721 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:38.211811 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260810 23:06:38.212198 19506 catalog_manager.cc:4584] TS cb4ef206029146d299745ce884c317ad (127.14.75.1:41115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58fb15dbdd0d4d19ba5acf772f5c1793: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41115: connect: Connection refused (error 111)
I20260810 23:06:38.217836 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19856
I20260810 23:06:38.224388 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 19490
2026-08-10T23:06:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1138 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:06:38.247319 14636 mini_oidc.cc:152] Starting JWKS server
I20260810 23:06:38.249081 14636 webserver.cc:460] Webserver started at https://127.0.0.1:45437/ using document root <none> and password file <none>
I20260810 23:06:38.249270 14636 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:06:38.249603 14636 webserver.cc:460] Webserver started at http://127.0.0.1:43317/ using document root <none> and password file <none>
2026-08-10T23:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:38Z Disabled control of system clock
I20260810 23:06:38.259523 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41377
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45437/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/testchainca.cert with env {}
W20260810 23:06:38.371145 20046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:38.371421 20046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:38.371480 20046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:38.375418 20046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:38.375471 20046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:38.375486 20046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:38.375502 20046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:38.375517 20046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:06:38.375532 20046 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:06:38.375545 20046 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45437/jwks.json
I20260810 23:06:38.381309 20046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45437/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:38.382352 20046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:38.383296 20046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:38.391731 20054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:38.391698 20051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:38.391687 20052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:38.391969 20046 server_base.cc:1034] running on GCE node
I20260810 23:06:38.392683 20046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:38.393834 20046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:38.395027 20046 hybrid_clock.cc:648] HybridClock initialized: now 1786403198395017 us; error 25 us; skew 500 ppm
I20260810 23:06:38.396840 20046 webserver.cc:460] Webserver started at http://127.14.75.62:39087/ using document root <none> and password file <none>
I20260810 23:06:38.397480 20046 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:38.397576 20046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:38.397814 20046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:38.399946 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "016138476eda49ada30747682594f0ed"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "84b5846dc591dee07211cd01a9700954"
server_key_iv: "cb19a0dc44671d63943be7a53035964c"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.400559 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "016138476eda49ada30747682594f0ed"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "84b5846dc591dee07211cd01a9700954"
server_key_iv: "cb19a0dc44671d63943be7a53035964c"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.404413 20046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:06:38.407361 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:38.408573 20046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:38.408735 20046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "016138476eda49ada30747682594f0ed"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "84b5846dc591dee07211cd01a9700954"
server_key_iv: "cb19a0dc44671d63943be7a53035964c"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.408854 20046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:38.427162 20046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:38.427917 20046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:38.428128 20046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:38.435155 20046 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41377
I20260810 23:06:38.435169 20113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41377 every 8 connection(s)
I20260810 23:06:38.436220 20046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:38.439412 20114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:38.444916 20114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: Bootstrap starting.
I20260810 23:06:38.445894 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20046
I20260810 23:06:38.445999 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:38.446267 14636 external_mini_cluster.cc:1428] Setting key ae9fae47efbbf4ca583be72b835a237e
I20260810 23:06:38.447757 20114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:38.448504 20114 log.cc:824] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:38.450791 20114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: No bootstrap required, opened a new log
I20260810 23:06:38.455116 20114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } }
I20260810 23:06:38.455364 20114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:38.455443 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.447377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.14.75.62:41377)
0810 23:06:38.448090 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.448105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.448139 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:06:38.448802 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.448819 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.449251 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.449626 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:38.450009 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.450994 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.452205 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.452691 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.452803 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.453693 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:38.453698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:38.453711 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:38.454081 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:38.454134 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:38.454153 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:38.454244 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:38.454416 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 23:06:38.454706 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":161,"threads_started":1}
I20260810 23:06:38.455446 20114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016138476eda49ada30747682594f0ed, State: Initialized, Role: FOLLOWER
I20260810 23:06:38.456003 20114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } }
I20260810 23:06:38.456172 20114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:38.456248 20114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:38.456342 20114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:38.457319 20114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } }
I20260810 23:06:38.457712 20114 leader_election.cc:302] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 016138476eda49ada30747682594f0ed; no voters:
I20260810 23:06:38.458069 20114 leader_election.cc:288] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:38.458197 20119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:38.458364 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:38.458417 20119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [term 1 LEADER]: Becoming Leader. State: Replica: 016138476eda49ada30747682594f0ed, State: Running, Role: LEADER
I20260810 23:06:38.458715 20119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } }
I20260810 23:06:38.459446 20114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:38.459908 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 016138476eda49ada30747682594f0ed. Latest consensus state: current_term: 1 leader_uuid: "016138476eda49ada30747682594f0ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } } }
I20260810 23:06:38.460037 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:38.460420 20120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "016138476eda49ada30747682594f0ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "016138476eda49ada30747682594f0ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41377 } } }
I20260810 23:06:38.460542 20120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed [sys.catalog]: This master's current role is: LEADER
W20260810 23:06:38.464391 20137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:38.464486 20137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:38.464561 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:38.465269 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:38.470979 20123 catalog_manager.cc:1347] Generated new cluster ID: 38a4695bcd8544e287b8fc5a2d6af647
I20260810 23:06:38.471064 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:38.509325 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:38.510317 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:38.518975 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 016138476eda49ada30747682594f0ed: Generated new TSK 0
I20260810 23:06:38.519663 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:06:38.553197 20147 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:06:38.553371 20147 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260810 23:06:38.554927 20146 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:38.529142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41377 (local address 127.0.0.1:51306)
0810 23:06:38.529676 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:38.529691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:38.529825 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:38.530164 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:38.530168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:38.530176 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:06:38.530310 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:38.530317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.531047 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.531051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:38.532070 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:38.532081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.532520 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.532525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:38.532624 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.532685 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:06:38.553558 (+ 20873us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:06:38.553771 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.75.62:41377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":139,"threads_started":1}
I20260810 23:06:38.555557 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.529327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.14.75.62:41377)
0810 23:06:38.529477 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.529482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.529772 (+ 290us) server_negotiation.cc:407] Connection header received
0810 23:06:38.529988 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.529990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.530036 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.530120 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:06:38.530418 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.530922 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.532185 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.532404 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.532479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.532873 (+ 394us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:06:38.553214 (+ 20341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:06:38.555373 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51306: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.75.62:41377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45437/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:06:38.555811 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20046
2026-08-10T23:06:38Z chronyd exiting
I20260810 23:06:38.594352 14636 test_util.cc:175] -----------------------------------------------
I20260810 23:06:38.594444 14636 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786403171263156-14636-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:06:38.595556 14636 mini_oidc.cc:152] Starting JWKS server
I20260810 23:06:38.596396 14636 webserver.cc:460] Webserver started at https://127.0.0.1:35601/ using document root <none> and password file <none>
I20260810 23:06:38.596477 14636 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:06:38.596732 14636 webserver.cc:460] Webserver started at http://127.0.0.1:34925/ using document root <none> and password file <none>
2026-08-10T23:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:38Z Disabled control of system clock
I20260810 23:06:38.606490 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45821
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:45821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35601/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/testchainca.cert with env {}
W20260810 23:06:38.715286 20153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:38.715605 20153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:38.715680 20153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:38.719758 20153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:38.719877 20153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:38.719906 20153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:38.719944 20153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:38.720007 20153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:06:38.720046 20153 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:06:38.720103 20153 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35601/jwks.json
I20260810 23:06:38.725931 20153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:45821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35601/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:38.727075 20153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:38.728116 20153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:38.736697 20161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:38.736685 20158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:38.736685 20159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:38.737090 20153 server_base.cc:1034] running on GCE node
I20260810 23:06:38.737741 20153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:38.738874 20153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:38.740068 20153 hybrid_clock.cc:648] HybridClock initialized: now 1786403198740050 us; error 37 us; skew 500 ppm
I20260810 23:06:38.741997 20153 webserver.cc:460] Webserver started at http://127.14.75.62:39753/ using document root <none> and password file <none>
I20260810 23:06:38.742584 20153 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:38.742678 20153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:38.742913 20153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:38.745031 20153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "a9788a65090c426483c11c4c83e04f64"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "8f1510e2fe49a38c97f83db14f1d76e7"
server_key_iv: "0fe1107079495595cb64a4c2304ed8f8"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.745711 20153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "a9788a65090c426483c11c4c83e04f64"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "8f1510e2fe49a38c97f83db14f1d76e7"
server_key_iv: "0fe1107079495595cb64a4c2304ed8f8"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.749356 20153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:06:38.751695 20168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:38.752830 20153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:06:38.752957 20153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "a9788a65090c426483c11c4c83e04f64"
format_stamp: "Formatted at 2026-08-10 23:06:38 on dist-test-slave-srgv"
server_key: "8f1510e2fe49a38c97f83db14f1d76e7"
server_key_iv: "0fe1107079495595cb64a4c2304ed8f8"
server_key_version: "encryptionkey@0"
I20260810 23:06:38.753070 20153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:38.764932 20153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:38.765632 20153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:38.765856 20153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:38.772784 20153 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:45821
I20260810 23:06:38.772797 20221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:45821 every 8 connection(s)
I20260810 23:06:38.773784 20153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:38.776908 20223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:38.782727 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20153
I20260810 23:06:38.782794 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64: Bootstrap starting.
I20260810 23:06:38.782866 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:38.783181 14636 external_mini_cluster.cc:1428] Setting key a53f3ac8d46389a6bdd2179b65375ccd
I20260810 23:06:38.785663 20223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:38.786576 20223 log.cc:824] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:38.788545 20223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64: No bootstrap required, opened a new log
I20260810 23:06:38.792490 20223 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } }
I20260810 23:06:38.792737 20223 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:38.792829 20223 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9788a65090c426483c11c4c83e04f64, State: Initialized, Role: FOLLOWER
I20260810 23:06:38.793246 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.784651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.14.75.62:45821)
0810 23:06:38.785361 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.785382 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.785453 (+ 71us) server_negotiation.cc:407] Connection header received
0810 23:06:38.786378 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.786405 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.786913 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.787308 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:38.787743 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.788783 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.789778 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.790250 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.790435 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.791403 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:38.791408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:38.791420 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:38.791805 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:38.791860 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:38.791880 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:38.791993 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:38.792150 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 23:06:38.792420 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260810 23:06:38.793321 20223 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } }
I20260810 23:06:38.793515 20223 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:38.793579 20223 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:38.793668 20223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:38.794760 20223 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } }
I20260810 23:06:38.795156 20223 leader_election.cc:302] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9788a65090c426483c11c4c83e04f64; no voters:
I20260810 23:06:38.795521 20223 leader_election.cc:288] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:38.795616 20228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:38.795838 20228 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [term 1 LEADER]: Becoming Leader. State: Replica: a9788a65090c426483c11c4c83e04f64, State: Running, Role: LEADER
I20260810 23:06:38.796137 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:38.796151 20228 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } }
I20260810 23:06:38.796803 20223 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:38.797533 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9788a65090c426483c11c4c83e04f64. Latest consensus state: current_term: 1 leader_uuid: "a9788a65090c426483c11c4c83e04f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } } }
I20260810 23:06:38.797678 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:38.798355 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9788a65090c426483c11c4c83e04f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9788a65090c426483c11c4c83e04f64" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45821 } } }
I20260810 23:06:38.798457 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:38.799213 20235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:38.801862 20235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:38.806874 20235 catalog_manager.cc:1347] Generated new cluster ID: b664d2a52c114c658c53098117ce7150
I20260810 23:06:38.806943 20235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:38.818411 20235 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:38.819640 20235 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:38.827819 20235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9788a65090c426483c11c4c83e04f64: Generated new TSK 0
I20260810 23:06:38.828511 20235 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:06:38.865329 20253 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:06:38.865450 20253 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:06:38.866102 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:38.842608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.14.75.62:45821)
0810 23:06:38.842757 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:38.842762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:38.842956 (+ 194us) server_negotiation.cc:407] Connection header received
0810 23:06:38.843118 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:38.843121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:38.843169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:38.843264 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:06:38.843520 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.844066 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.845316 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.845601 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.845682 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.846024 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:06:38.865726 (+ 19702us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:06:38.865978 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 23:06:38.866195 20226 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 23:06:38.866326 20252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:38.842518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45821 (local address 127.0.0.1:53376)
0810 23:06:38.842859 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:38.842875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:38.842998 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:38.843290 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:38.843294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:38.843302 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:06:38.843414 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:38.843421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.844189 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.844193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:38.845192 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:38.845203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:38.845731 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:38.845735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:38.845827 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:38.845856 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:06:38.865951 (+ 20095us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:06:38.866145 (+ 194us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.75.62:45821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260810 23:06:38.867051 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20153
2026-08-10T23:06:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:06:38.907950 14636 mini_oidc.cc:152] Starting JWKS server
I20260810 23:06:38.908871 14636 webserver.cc:460] Webserver started at https://127.0.0.1:45427/ using document root <none> and password file <none>
I20260810 23:06:38.909014 14636 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:06:38.909292 14636 webserver.cc:460] Webserver started at http://127.0.0.1:35117/ using document root <none> and password file <none>
2026-08-10T23:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:38Z Disabled control of system clock
I20260810 23:06:38.919011 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36061
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:36061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:06:39.030334 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:39.030619 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:39.030658 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:39.034694 20259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:39.034762 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:39.034777 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:39.034795 20259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:39.034811 20259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:06:39.034826 20259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:06:39.034853 20259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45427/jwks.json
I20260810 23:06:39.039800 20259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:36061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:39.040892 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:39.042260 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:39.050889 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:39.051139 20259 server_base.cc:1034] running on GCE node
W20260810 23:06:39.050889 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:39.050928 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:39.051903 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:39.053385 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:39.054633 20259 hybrid_clock.cc:648] HybridClock initialized: now 1786403199054601 us; error 49 us; skew 500 ppm
I20260810 23:06:39.057245 20259 webserver.cc:460] Webserver started at http://127.14.75.62:36157/ using document root <none> and password file <none>
I20260810 23:06:39.058032 20259 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:39.058154 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:39.058414 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:39.061182 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "c30c8868d5b545ff99b511487e3d690a"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "cc3a2283a389b94a2a2fc7eda13037b8"
server_key_iv: "f7d2bea357fbb643f32d12f4b3ea61ff"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.061981 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "c30c8868d5b545ff99b511487e3d690a"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "cc3a2283a389b94a2a2fc7eda13037b8"
server_key_iv: "f7d2bea357fbb643f32d12f4b3ea61ff"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.065891 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:39.068326 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:39.069625 20259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:39.069726 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "c30c8868d5b545ff99b511487e3d690a"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "cc3a2283a389b94a2a2fc7eda13037b8"
server_key_iv: "f7d2bea357fbb643f32d12f4b3ea61ff"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.069820 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:39.084769 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:39.085510 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:39.085675 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:39.092267 20259 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:36061
I20260810 23:06:39.092280 20330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:36061 every 8 connection(s)
I20260810 23:06:39.093223 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:39.094851 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20259
I20260810 23:06:39.094983 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:39.095301 14636 external_mini_cluster.cc:1428] Setting key e61008a989a393600005edc78b1a1d92
I20260810 23:06:39.096895 20331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:39.102888 20331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a: Bootstrap starting.
I20260810 23:06:39.104696 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:39.096695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.14.75.62:36061)
0810 23:06:39.097362 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:39.097377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:39.097466 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:06:39.098113 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:39.098129 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:39.098600 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:39.098983 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:39.099441 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.100430 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.101339 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.101868 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.102044 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.102988 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:39.102997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:39.103014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:39.103408 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:39.103464 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:39.103484 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:39.103601 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:39.103737 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:06:39.103992 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":147,"threads_started":1}
I20260810 23:06:39.105295 20331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:39.106040 20331 log.cc:824] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:39.107493 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:39.108229 20331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a: No bootstrap required, opened a new log
I20260810 23:06:39.111975 20331 raft_consensus.cc:348] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } }
I20260810 23:06:39.112193 20331 raft_consensus.cc:374] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:39.112246 20331 raft_consensus.cc:729] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c8868d5b545ff99b511487e3d690a, State: Initialized, Role: FOLLOWER
I20260810 23:06:39.112655 20331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } }
I20260810 23:06:39.112780 20331 raft_consensus.cc:388] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:39.112828 20331 raft_consensus.cc:482] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:39.112896 20331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:39.113857 20331 raft_consensus.cc:504] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } }
I20260810 23:06:39.114147 20331 leader_election.cc:302] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c30c8868d5b545ff99b511487e3d690a; no voters:
I20260810 23:06:39.114413 20331 leader_election.cc:288] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:39.114558 20336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:39.114786 20336 raft_consensus.cc:686] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [term 1 LEADER]: Becoming Leader. State: Replica: c30c8868d5b545ff99b511487e3d690a, State: Running, Role: LEADER
I20260810 23:06:39.115101 20336 consensus_queue.cc:237] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } }
I20260810 23:06:39.115724 20331 sys_catalog.cc:564] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:39.117065 20337 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c30c8868d5b545ff99b511487e3d690a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } } }
I20260810 23:06:39.117097 20338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c30c8868d5b545ff99b511487e3d690a. Latest consensus state: current_term: 1 leader_uuid: "c30c8868d5b545ff99b511487e3d690a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c30c8868d5b545ff99b511487e3d690a" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36061 } } }
I20260810 23:06:39.117233 20337 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:39.117236 20338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:39.117619 20345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:39.120239 20345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:39.125921 20345 catalog_manager.cc:1347] Generated new cluster ID: 8fc53f639ffb44849cce3dc8fb5902d0
I20260810 23:06:39.125996 20345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:39.132144 20345 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:39.133339 20345 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:39.140025 20345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c30c8868d5b545ff99b511487e3d690a: Generated new TSK 0
I20260810 23:06:39.140820 20345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:39.161003 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:39.155771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.14.75.62:36061)
0810 23:06:39.155918 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:39.155923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:39.156074 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:06:39.156222 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:39.156227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:39.156290 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:39.156414 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:06:39.156627 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.157483 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.158377 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.158734 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.158837 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.159136 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:06:39.160599 (+ 1463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:06:39.160842 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:06:39.161082 20334 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:06:39.162698 20362 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:39.155643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:36061 (local address 127.0.0.1:41160)
0810 23:06:39.155977 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:39.155990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:39.156098 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:39.156416 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:39.156418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:39.156424 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:06:39.156523 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:39.156527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.157628 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.157631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:39.158258 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:39.158265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.158862 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.158865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:39.158930 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.158953 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:06:39.160781 (+ 1828us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:06:39.162520 (+ 1739us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.75.62:36061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45427/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1541,"thread_start_us":110,"threads_started":1}
I20260810 23:06:39.163261 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20259
E20260810 23:06:39.175246 20363 webserver.cc:533] Webserver: sslize failed:
E20260810 23:06:39.175333 20363 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T23:06:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:39Z Disabled control of system clock
I20260810 23:06:39.215194 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46391
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46391
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/testchainca.cert with env {}
W20260810 23:06:39.327798 20367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:39.328087 20367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:39.328130 20367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:39.332397 20367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:39.332474 20367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:39.332490 20367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:39.332507 20367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:39.332523 20367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:06:39.332584 20367 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:06:39.332603 20367 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:06:39.338192 20367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:39.339334 20367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:39.340412 20367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:39.348853 20373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:39.348927 20372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:39.348853 20375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:39.349117 20367 server_base.cc:1034] running on GCE node
I20260810 23:06:39.349875 20367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:39.350999 20367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:39.352206 20367 hybrid_clock.cc:648] HybridClock initialized: now 1786403199352188 us; error 37 us; skew 500 ppm
I20260810 23:06:39.354219 20367 webserver.cc:460] Webserver started at http://127.14.75.62:43985/ using document root <none> and password file <none>
I20260810 23:06:39.354811 20367 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:39.354908 20367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:39.355129 20367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:39.357266 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "0a6ef0a343324e288593506669a42a8e"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "7fd43de78e1108231db393e40281c57f"
server_key_iv: "43d97f7f9762480f98763423b1696239"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.357941 20367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "0a6ef0a343324e288593506669a42a8e"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "7fd43de78e1108231db393e40281c57f"
server_key_iv: "43d97f7f9762480f98763423b1696239"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.361660 20367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:06:39.364084 20381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:39.365301 20367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:39.365491 20367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "0a6ef0a343324e288593506669a42a8e"
format_stamp: "Formatted at 2026-08-10 23:06:39 on dist-test-slave-srgv"
server_key: "7fd43de78e1108231db393e40281c57f"
server_key_iv: "43d97f7f9762480f98763423b1696239"
server_key_version: "encryptionkey@0"
I20260810 23:06:39.365621 20367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:39.383020 20367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:39.383786 20367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:39.383991 20367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:39.390866 20367 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46391
I20260810 23:06:39.390882 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46391 every 8 connection(s)
I20260810 23:06:39.391808 20367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:39.394656 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:39.400182 20436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e: Bootstrap starting.
I20260810 23:06:39.401573 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20367
I20260810 23:06:39.401721 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:39.402011 14636 external_mini_cluster.cc:1428] Setting key 55fe17cda43b22093799b9ce28abef55
I20260810 23:06:39.403251 20436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:39.404186 20436 log.cc:824] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:39.406493 20436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e: No bootstrap required, opened a new log
I20260810 23:06:39.410794 20436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } }
I20260810 23:06:39.411123 20436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:39.411185 20436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6ef0a343324e288593506669a42a8e, State: Initialized, Role: FOLLOWER
I20260810 23:06:39.411657 20436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } }
I20260810 23:06:39.411839 20436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:39.411893 20436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:39.412009 20436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:39.412390 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:39.403130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.14.75.62:46391)
0810 23:06:39.403856 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:39.403874 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:39.403920 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:06:39.404768 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:39.404786 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:39.405185 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:39.405576 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:39.406012 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.407437 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.408383 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.409099 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.409280 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.410181 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:39.410187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:39.410205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:39.410711 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:39.410789 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:39.410819 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:39.410922 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:39.411058 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:06:39.411338 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
I20260810 23:06:39.413072 20436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } }
I20260810 23:06:39.413471 20436 leader_election.cc:302] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a6ef0a343324e288593506669a42a8e; no voters:
I20260810 23:06:39.413879 20436 leader_election.cc:288] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:39.414017 20441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:39.414265 20441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [term 1 LEADER]: Becoming Leader. State: Replica: 0a6ef0a343324e288593506669a42a8e, State: Running, Role: LEADER
I20260810 23:06:39.414595 20441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } }
I20260810 23:06:39.415148 20436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:39.415721 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:39.415804 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a6ef0a343324e288593506669a42a8e. Latest consensus state: current_term: 1 leader_uuid: "0a6ef0a343324e288593506669a42a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } } }
I20260810 23:06:39.415925 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:39.416245 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a6ef0a343324e288593506669a42a8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6ef0a343324e288593506669a42a8e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46391 } } }
I20260810 23:06:39.416347 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:39.416931 20447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:39.420437 20447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:39.426415 20447 catalog_manager.cc:1347] Generated new cluster ID: c78d071f549d41f8b837f0c6bc60e452
I20260810 23:06:39.426499 20447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:39.432721 20447 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:39.433861 20447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:39.459162 20447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a6ef0a343324e288593506669a42a8e: Generated new TSK 0
I20260810 23:06:39.460041 20447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:39.531560 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:39.468828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.14.75.62:46391)
0810 23:06:39.468979 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:39.468985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:39.469286 (+ 301us) server_negotiation.cc:407] Connection header received
0810 23:06:39.469354 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:39.469356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:39.469401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:39.469543 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:06:39.469834 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.470418 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.471618 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.471838 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.471909 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.472242 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:06:39.531159 (+ 58917us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:06:39.531442 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:06:39.531675 20439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:06:39.531874 20465 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:39.468777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46391 (local address 127.0.0.1:58962)
0810 23:06:39.469086 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:39.469101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:39.469219 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:39.469562 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:39.469566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:39.469574 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:06:39.469729 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:39.469736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.470536 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.470541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:39.471500 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:39.471510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:39.471952 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:39.471956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:39.472052 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:39.472079 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:06:39.531427 (+ 59348us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:06:39.531682 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.75.62:46391: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260810 23:06:39.532706 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20367
2026-08-10T23:06:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (347 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (125 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:39 dist-test-slave-srgv krb5kdc[20477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:39 dist-test-slave-srgv krb5kdc[20477](info): set up 2 sockets
Aug 10 23:06:39 dist-test-slave-srgv krb5kdc[20477](info): commencing operation
krb5kdc: starting...
W20260810 23:06:40.840766 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:40 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:40.984336 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40957
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:41.097436 20491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:41.097740 20491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:41.097782 20491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:41.101969 20491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:41.102033 20491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:41.102052 20491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:41.102069 20491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:41.102085 20491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:41.107525 20491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:41.108631 20491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:41.109740 20491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:41.118229 20496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.118257 20499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.118453 20497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:41.118481 20491 server_base.cc:1034] running on GCE node
I20260810 23:06:41.119249 20491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:41.120301 20491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:41.121532 20491 hybrid_clock.cc:648] HybridClock initialized: now 1786403201121518 us; error 38 us; skew 500 ppm
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.127923 20491 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:41.129053 20491 webserver.cc:460] Webserver started at http://127.14.75.62:38345/ using document root <none> and password file <none>
I20260810 23:06:41.129714 20491 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:41.129782 20491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:41.130061 20491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:41.132287 20491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "652920481b974b499ceb4df843d326f8"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "e533014cfe9b3422befa903e8c6cbecb"
server_key_iv: "11f9185b785392b81f0e2f27313e6065"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.132898 20491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "652920481b974b499ceb4df843d326f8"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "e533014cfe9b3422befa903e8c6cbecb"
server_key_iv: "11f9185b785392b81f0e2f27313e6065"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.136842 20491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:41.139410 20507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.140594 20491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:41.140743 20491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "652920481b974b499ceb4df843d326f8"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "e533014cfe9b3422befa903e8c6cbecb"
server_key_iv: "11f9185b785392b81f0e2f27313e6065"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.140851 20491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:41.162638 20491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:41.165738 20491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:41.165968 20491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:41.173646 20491 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:40957
I20260810 23:06:41.173660 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:40957 every 8 connection(s)
I20260810 23:06:41.174647 20491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:41.177541 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:41.182024 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20491
I20260810 23:06:41.182173 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:41.182441 14636 external_mini_cluster.cc:1428] Setting key cf192b66d4b11e0894d0ba14a64694e1
I20260810 23:06:41.183359 20561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: Bootstrap starting.
I20260810 23:06:41.185907 20561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:41.186905 20561 log.cc:824] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:41.189222 20561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: No bootstrap required, opened a new log
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:41.194217 20561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } }
I20260810 23:06:41.194496 20561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:41.194591 20561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652920481b974b499ceb4df843d326f8, State: Initialized, Role: FOLLOWER
I20260810 23:06:41.195097 20561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } }
I20260810 23:06:41.195254 20561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:41.195344 20561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:41.195446 20561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:41.196421 20561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } }
I20260810 23:06:41.196828 20561 leader_election.cc:302] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 652920481b974b499ceb4df843d326f8; no voters:
I20260810 23:06:41.197227 20561 leader_election.cc:288] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:41.197525 20567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:41.197882 20567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [term 1 LEADER]: Becoming Leader. State: Replica: 652920481b974b499ceb4df843d326f8, State: Running, Role: LEADER
I20260810 23:06:41.198228 20567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } }
I20260810 23:06:41.198575 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.183909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36906 (local address 127.14.75.62:40957)
0810 23:06:41.184407 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.184422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.184459 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:41.185097 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.185113 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.185519 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.185894 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.187092 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.188280 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.189477 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.189942 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.190063 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.192960 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.192966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.192987 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.193014 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.195019 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.195584 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.195590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.195597 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.195728 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.196057 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.196063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.196067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.196504 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.196705 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.197048 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.197398 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":144,"threads_started":1}
I20260810 23:06:41.198649 20561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:41.200062 20569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652920481b974b499ceb4df843d326f8. Latest consensus state: current_term: 1 leader_uuid: "652920481b974b499ceb4df843d326f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } } }
I20260810 23:06:41.200194 20569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:41.200471 20568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "652920481b974b499ceb4df843d326f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "652920481b974b499ceb4df843d326f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40957 } } }
I20260810 23:06:41.200557 20568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8 [sys.catalog]: This master's current role is: LEADER
W20260810 23:06:41.203693 20582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:41.203783 20582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:41.203855 20578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:41.204767 20578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:41.210664 20578 catalog_manager.cc:1347] Generated new cluster ID: 0e08c16633d0419598ae43bc3c3c5302
I20260810 23:06:41.210763 20578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:41.225180 20578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:41.226584 20578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:41.233062 20578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 652920481b974b499ceb4df843d326f8: Generated new TSK 0
I20260810 23:06:41.233866 20578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:41.273568 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:41.390360 20589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:41.390655 20589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:41.390702 20589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:41.395007 20589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:41.395088 20589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:41.395197 20589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:41.401361 20589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:41.402577 20589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:41.403659 20589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:41.413748 20597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.413779 20594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.413751 20595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:41.414073 20589 server_base.cc:1034] running on GCE node
I20260810 23:06:41.414534 20589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:41.415302 20589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:41.416513 20589 hybrid_clock.cc:648] HybridClock initialized: now 1786403201416502 us; error 33 us; skew 500 ppm
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.419718 20589 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:41.420739 20589 webserver.cc:460] Webserver started at http://127.14.75.1:33965/ using document root <none> and password file <none>
I20260810 23:06:41.421372 20589 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:41.421531 20589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:41.421783 20589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:41.423921 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "63067e1b62ee4efc852c117e834c86a1"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "25cd620da041b3292388ac136c5ce157"
server_key_iv: "fb3635929134e24f8cd20c00d524cb8b"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.424543 20589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "63067e1b62ee4efc852c117e834c86a1"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "25cd620da041b3292388ac136c5ce157"
server_key_iv: "fb3635929134e24f8cd20c00d524cb8b"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.428511 20589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:41.431208 20604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.432507 20589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:41.432658 20589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "63067e1b62ee4efc852c117e834c86a1"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "25cd620da041b3292388ac136c5ce157"
server_key_iv: "fb3635929134e24f8cd20c00d524cb8b"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.432788 20589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:41.447618 20589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:41.450855 20589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:41.451090 20589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:41.451720 20589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:41.452571 20589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:41.452644 20589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.452714 20589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:41.452766 20589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.461656 20589 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:37221
I20260810 23:06:41.461684 20722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:37221 every 8 connection(s)
I20260810 23:06:41.462628 20589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:41.471040 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20589
I20260810 23:06:41.471185 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:41.471505 14636 external_mini_cluster.cc:1428] Setting key 0fe748278a6b990309a286394676cb7d
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:41.477042 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.464425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:33827 (local address 127.14.75.62:40957)
0810 23:06:41.464565 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.464569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.465387 (+ 818us) server_negotiation.cc:407] Connection header received
0810 23:06:41.466770 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.466774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.466837 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.466945 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.468134 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.468679 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.469845 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.470087 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.470199 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.473300 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.473303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.473305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.473329 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.474912 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.475460 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.475463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.475465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.475551 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.475906 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.475911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.475912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.476105 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.476224 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.476743 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.476877 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:41.477938 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.464736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40957 (local address 127.14.75.1:33827)
0810 23:06:41.465220 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:41.465263 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:41.466550 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:41.467122 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:41.467129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:41.467499 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:41.467974 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.467988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.468810 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.468814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.469708 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.469718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.470242 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.470246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.470412 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.471153 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:41.471178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:41.473131 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:41.475045 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.475052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.475069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.475349 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.475646 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.475649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.475651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.475807 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.476216 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:41.476223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:41.476489 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 23:06:41.476686 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 23:06:41.477037 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260810 23:06:41.479233 20723 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40957
I20260810 23:06:41.479506 20723 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:41.480132 20723 heartbeater.cc:511] Master 127.14.75.62:40957 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:41.481932 20524 ts_manager.cc:194] Registered new tserver with Master: 63067e1b62ee4efc852c117e834c86a1 (127.14.75.1:37221)
I20260810 23:06:41.483180 20524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:33827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:41.501529 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:41.615586 20728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:41.615855 20728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:41.615906 20728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:41.620098 20728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:41.620168 20728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:41.620276 20728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:41.626343 20728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:41.627472 20728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:41.628536 20728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:41.638706 20728 server_base.cc:1034] running on GCE node
W20260810 23:06:41.638607 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.638612 20733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.638635 20734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:41.639315 20728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:41.639953 20728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:41.641140 20728 hybrid_clock.cc:648] HybridClock initialized: now 1786403201641115 us; error 55 us; skew 500 ppm
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.643939 20728 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:41.644974 20728 webserver.cc:460] Webserver started at http://127.14.75.2:36267/ using document root <none> and password file <none>
I20260810 23:06:41.645646 20728 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:41.645744 20728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:41.645975 20728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:41.648052 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "6b5b933a836f4c48b49faeb9f450a91d"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "d152f6fe884ba65da6ba2deb3b3b81ab"
server_key_iv: "8ae7812f7cda5de8f7abee1904ffbd0d"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.648643 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "6b5b933a836f4c48b49faeb9f450a91d"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "d152f6fe884ba65da6ba2deb3b3b81ab"
server_key_iv: "8ae7812f7cda5de8f7abee1904ffbd0d"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.652428 20728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:41.654984 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.656191 20728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:41.656339 20728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "6b5b933a836f4c48b49faeb9f450a91d"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "d152f6fe884ba65da6ba2deb3b3b81ab"
server_key_iv: "8ae7812f7cda5de8f7abee1904ffbd0d"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.656462 20728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:41.669090 20728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:41.672276 20728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:41.672506 20728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:41.673146 20728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:41.674077 20728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:41.674153 20728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.674230 20728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:41.674274 20728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.683214 20728 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:44673
I20260810 23:06:41.683298 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:44673 every 8 connection(s)
I20260810 23:06:41.684216 20728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:41.688262 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20728
I20260810 23:06:41.688401 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:41.688688 14636 external_mini_cluster.cc:1428] Setting key fb78dcd4a2618c778c9007c11111ab81
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:41.700377 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.686268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:59185 (local address 127.14.75.62:40957)
0810 23:06:41.686450 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.686455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.687409 (+ 954us) server_negotiation.cc:407] Connection header received
0810 23:06:41.689023 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.689027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.689091 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.689197 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.690684 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.691231 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.692715 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.692975 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.693108 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.696221 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.696225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.696227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.696254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.698009 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.698587 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.698590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.698592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.698678 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.699125 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.699130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.699136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.699323 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.699436 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.700062 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.700202 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:06:41.701490 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.686665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40957 (local address 127.14.75.2:59185)
0810 23:06:41.687226 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:41.687272 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:41.688754 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:41.689573 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:41.689583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:41.689993 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:41.690512 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.690527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.691383 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.691387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.692400 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.692411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.693164 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.693168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.693469 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.694171 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:41.694197 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:41.696024 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:41.698166 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.698172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.698189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.698466 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.698775 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.698779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.698781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.698998 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.699433 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:41.699441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:41.699776 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 23:06:41.699976 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:06:41.700323 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:06:41.702821 20860 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40957
I20260810 23:06:41.703099 20860 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:41.703789 20860 heartbeater.cc:511] Master 127.14.75.62:40957 requested a full tablet report, sending...
I20260810 23:06:41.705106 20524 ts_manager.cc:194] Registered new tserver with Master: 6b5b933a836f4c48b49faeb9f450a91d (127.14.75.2:44673)
I20260810 23:06:41.705782 20524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:59185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:41.720060 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:41.833863 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:41.834160 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:41.834206 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:41.838845 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:41.838914 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:41.839010 20865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:41.844673 20865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:41.845947 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:41.846985 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:41.856978 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.856979 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:41.857004 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:41.857474 20865 server_base.cc:1034] running on GCE node
I20260810 23:06:41.857899 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:41.858500 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:41.859661 20865 hybrid_clock.cc:648] HybridClock initialized: now 1786403201859648 us; error 28 us; skew 500 ppm
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.862561 20865 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:41.863396 20865 webserver.cc:460] Webserver started at http://127.14.75.3:44251/ using document root <none> and password file <none>
I20260810 23:06:41.863932 20865 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:41.863998 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:41.864194 20865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:41.866272 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "98d8825de0464997b2e5a619581ff572"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "c8f0435cba4632a76a3bae66b7628b6b"
server_key_iv: "37a27725c965bd467bd970f916ef09b5"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.866845 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "98d8825de0464997b2e5a619581ff572"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "c8f0435cba4632a76a3bae66b7628b6b"
server_key_iv: "37a27725c965bd467bd970f916ef09b5"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.870741 20865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:41.873162 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.874567 20865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:41.874719 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "98d8825de0464997b2e5a619581ff572"
format_stamp: "Formatted at 2026-08-10 23:06:41 on dist-test-slave-srgv"
server_key: "c8f0435cba4632a76a3bae66b7628b6b"
server_key_iv: "37a27725c965bd467bd970f916ef09b5"
server_key_version: "encryptionkey@0"
I20260810 23:06:41.874842 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:41.902316 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:41.905284 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:41.905544 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:41.906177 20865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:41.907099 20865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:41.907171 20865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.907246 20865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:41.907291 20865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:41.916150 20865 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:33651
I20260810 23:06:41.916168 20995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:33651 every 8 connection(s)
I20260810 23:06:41.917148 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:41.927419 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 20865
I20260810 23:06:41.927534 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:41.927922 14636 external_mini_cluster.cc:1428] Setting key e2da6976906c188d4011844c9d48a141
I20260810 23:06:41.931507 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.919024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:43633 (local address 127.14.75.62:40957)
0810 23:06:41.919160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.919164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.920030 (+ 866us) server_negotiation.cc:407] Connection header received
0810 23:06:41.921447 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.921450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.921511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.921600 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.922768 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.923319 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.924464 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.924705 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.924820 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.927433 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.927436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.927438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.927463 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.929213 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.929856 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.929860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.929861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.929957 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.930404 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.930410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.930413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.930612 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.930731 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.931235 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.931351 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:06:41.932346 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.919413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40957 (local address 127.14.75.3:43633)
0810 23:06:41.919879 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:41.919922 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:41.921212 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:41.921772 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:41.921778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:41.922145 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:41.922608 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.922621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.923455 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.923463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.924337 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:41.924346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.924862 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.924866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:41.925031 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.925685 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:41.925701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:41.927239 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:41.929336 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.929342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.929363 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.929682 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.930065 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:41.930067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:41.930068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:41.930290 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:41.930723 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:41.930729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:41.930990 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 23:06:41.931196 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 23:06:41.931532 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
I20260810 23:06:41.933492 20996 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40957
I20260810 23:06:41.933748 20996 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:41.934247 20996 heartbeater.cc:511] Master 127.14.75.62:40957 requested a full tablet report, sending...
I20260810 23:06:41.935236 20524 ts_manager.cc:194] Registered new tserver with Master: 98d8825de0464997b2e5a619581ff572 (127.14.75.3:33651)
I20260810 23:06:41.935835 20524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:43633
I20260810 23:06:41.942147 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/test-admin.keytab.
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.962674 14636 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:06:41.965848 21003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:41.964132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.14.75.1:37221)
0810 23:06:41.964429 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.964445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.964471 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:06:41.964633 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.964639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.964832 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.964986 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.965626 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44736: BlockingRecv error: recv got EOF from 127.0.0.1:44736 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
W20260810 23:06:41.965937 14636 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.967011 14636 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.967952 14636 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:41.977866 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.968839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.14.75.1:37221)
0810 23:06:41.968971 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.968977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.969070 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:06:41.969247 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.969249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.969290 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.969376 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.969959 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.970740 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.971589 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.971975 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.972097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.974434 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.974443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.974456 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.974480 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.975921 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.976396 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.976401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.976402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.976480 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.976779 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.976783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.976785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.977041 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.977307 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.977478 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.977631 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:06:41.982369 21003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:41.981264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.14.75.1:37221)
0810 23:06:41.981389 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.981393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.981538 (+ 145us) server_negotiation.cc:407] Connection header received
0810 23:06:41.981678 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.981681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.981727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.981809 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.982233 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44760: BlockingRecv error: recv got EOF from 127.0.0.1:44760 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 23:06:41.982725 14636 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc)
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.983754 14636 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.984659 14636 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:41.993546 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.985556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.14.75.1:37221)
0810 23:06:41.985682 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.985688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.985749 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:06:41.985914 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.985917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.985965 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.986049 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.986580 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.987105 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.987916 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:41.988175 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:41.988287 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:41.990584 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:41.990588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:41.990591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:41.990625 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:41.991820 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.992256 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.992259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.992261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.992345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:41.992620 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:41.992624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:41.992626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:41.992810 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:41.992980 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:41.993227 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 23:06:41.993344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 23:06:41.996181 21003 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:41.995308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.14.75.1:37221)
0810 23:06:41.995447 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.995450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.995467 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:41.995590 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.995593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.995626 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.995698 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:41.996092 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44784: BlockingRecv error: recv got EOF from 127.0.0.1:44784 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 23:06:41.996302 14636 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786403171263156-14636-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.997370 14636 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:41.998340 14636 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:06:41 dist-test-slave-srgv krb5kdc[20477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:06:42.006783 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:41.999199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.14.75.1:37221)
0810 23:06:41.999327 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:41.999332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:41.999432 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:06:41.999595 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:41.999599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:41.999640 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:41.999733 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:42.000230 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:42.000707 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:42.001550 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:42.001758 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:42.001832 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:42.004094 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:42.004097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:42.004099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:42.004120 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:42.005346 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:42.005816 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:42.005819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:42.005820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:42.005892 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:42.006116 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:42.006118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:42.006119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:42.006281 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:42.006388 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:42.006569 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:06:42.006658 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:42.007896 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20589
I20260810 23:06:42.015156 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20728
I20260810 23:06:42.021624 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20865
I20260810 23:06:42.027869 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 20491
2026-08-10T23:06:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2245 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:42 dist-test-slave-srgv krb5kdc[21016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:42 dist-test-slave-srgv krb5kdc[21016](info): set up 2 sockets
Aug 10 23:06:42 dist-test-slave-srgv krb5kdc[21016](info): commencing operation
krb5kdc: starting...
W20260810 23:06:43.096107 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab.
Entry for principal oryx/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab.
I20260810 23:06:43.228552 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43701
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab
--principal=oryx/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:43.341238 21029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:43.341650 21029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:43.341738 21029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:43.345822 21029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:43.345929 21029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:43.345966 21029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:43.346004 21029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:43.346040 21029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:43.351563 21029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:43701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab
--principal=oryx/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:43.352733 21029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:43.353865 21029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:43.362385 21034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:43.362411 21035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:43.362792 21029 server_base.cc:1034] running on GCE node
W20260810 23:06:43.362411 21037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:43.363394 21029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:43.364548 21029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:43.365751 21029 hybrid_clock.cc:648] HybridClock initialized: now 1786403203365728 us; error 36 us; skew 500 ppm
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:43.368623 21029 init.cc:377] Logged in from keytab as oryx/127.14.75.62@KRBTEST.COM (short username oryx)
I20260810 23:06:43.369553 21029 webserver.cc:460] Webserver started at http://127.14.75.62:46087/ using document root <none> and password file <none>
I20260810 23:06:43.370085 21029 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:43.370143 21029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:43.370306 21029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:43.372314 21029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "cb837fb6159943aea1dfbdac2f5236d8"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "dc295fb77aceded8b383f086c93a136b"
server_key_iv: "dea8b35cc1022485f02f12798ec52946"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.372843 21029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "cb837fb6159943aea1dfbdac2f5236d8"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "dc295fb77aceded8b383f086c93a136b"
server_key_iv: "dea8b35cc1022485f02f12798ec52946"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.376950 21029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:06:43.379673 21044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.380914 21029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:43.381044 21029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "cb837fb6159943aea1dfbdac2f5236d8"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "dc295fb77aceded8b383f086c93a136b"
server_key_iv: "dea8b35cc1022485f02f12798ec52946"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.381158 21029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:43.400942 21029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:43.404512 21029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:43.404755 21029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:43.411938 21029 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:43701
I20260810 23:06:43.412225 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:43701 every 8 connection(s)
I20260810 23:06:43.412992 21029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:43.415596 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21029
I20260810 23:06:43.415762 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:43.415881 21099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:43.416110 14636 external_mini_cluster.cc:1428] Setting key f603759d50e4f4f299a9daace3103941
I20260810 23:06:43.423435 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Bootstrap starting.
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:43.426249 21099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:43.427235 21099 log.cc:824] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:43.429710 21099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: No bootstrap required, opened a new log
I20260810 23:06:43.431810 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:43.417537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.14.75.62:43701)
0810 23:06:43.417981 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:43.418001 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:43.418053 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:06:43.418811 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:43.418834 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:43.419254 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:43.419637 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:43.420328 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.421867 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.422702 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.423399 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.423569 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:43.426461 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:43.426471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:43.426491 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:43.426530 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:43.428528 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.428961 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.428967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.428973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.429107 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.429376 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.429379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.429380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.429922 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:43.430129 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:43.430319 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:06:43.430626 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260810 23:06:43.433525 21099 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } }
I20260810 23:06:43.433750 21099 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:43.433812 21099 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb837fb6159943aea1dfbdac2f5236d8, State: Initialized, Role: FOLLOWER
I20260810 23:06:43.434219 21099 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } }
I20260810 23:06:43.434372 21099 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:43.434460 21099 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:43.434577 21099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:43.435534 21099 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } }
I20260810 23:06:43.435880 21099 leader_election.cc:302] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb837fb6159943aea1dfbdac2f5236d8; no voters:
I20260810 23:06:43.436205 21099 leader_election.cc:288] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:43.436345 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:43.436584 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 LEADER]: Becoming Leader. State: Replica: cb837fb6159943aea1dfbdac2f5236d8, State: Running, Role: LEADER
I20260810 23:06:43.436964 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } }
I20260810 23:06:43.437610 21099 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:43.438793 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } }
I20260810 23:06:43.438791 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb837fb6159943aea1dfbdac2f5236d8. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } }
I20260810 23:06:43.438962 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:43.438962 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:43.439386 21113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:43.442234 21113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:43.447690 21113 catalog_manager.cc:1347] Generated new cluster ID: 587399e21b03460cab9966f7f29d0af3
I20260810 23:06:43.447772 21113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:43.456967 21113 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:43.458338 21113 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:43.465039 21113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Generated new TSK 0
I20260810 23:06:43.465905 21113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
I20260810 23:06:43.504819 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:43.622484 21126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:43.622799 21126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:43.622852 21126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:43.627063 21126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:43.627137 21126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:43.627198 21126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:43.632690 21126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:43.633952 21126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:43.634982 21126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:43.645476 21134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:43.645548 21131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:43.645565 21132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:43.645891 21126 server_base.cc:1034] running on GCE node
I20260810 23:06:43.646404 21126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:43.647141 21126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:43.648358 21126 hybrid_clock.cc:648] HybridClock initialized: now 1786403203648327 us; error 56 us; skew 500 ppm
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:43.651566 21126 init.cc:377] Logged in from keytab as oryx/127.14.75.1@KRBTEST.COM (short username oryx)
I20260810 23:06:43.652467 21126 webserver.cc:460] Webserver started at http://127.14.75.1:38793/ using document root <none> and password file <none>
I20260810 23:06:43.653043 21126 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:43.653108 21126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:43.653283 21126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:43.655777 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "ceb95e4ce6f8428d95cc08d447637af2"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "2151bb1c237a3849417bedbf15258b1b"
server_key_iv: "783033919cdb9b980dc19b49c80df941"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.656502 21126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb95e4ce6f8428d95cc08d447637af2"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "2151bb1c237a3849417bedbf15258b1b"
server_key_iv: "783033919cdb9b980dc19b49c80df941"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.660665 21126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:43.663335 21141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.664659 21126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:43.664769 21126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "ceb95e4ce6f8428d95cc08d447637af2"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "2151bb1c237a3849417bedbf15258b1b"
server_key_iv: "783033919cdb9b980dc19b49c80df941"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.664870 21126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:43.678158 21126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:43.681298 21126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:43.681562 21126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:43.682235 21126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:43.683152 21126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:43.683202 21126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.683275 21126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:43.683306 21126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.692607 21126 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:45961
I20260810 23:06:43.692631 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:45961 every 8 connection(s)
I20260810 23:06:43.693660 21126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:43.702426 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21126
I20260810 23:06:43.702602 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:43.702913 14636 external_mini_cluster.cc:1428] Setting key 0b7b9136095012636b51c7953f0fa131
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:43.709023 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:43.695654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:60225 (local address 127.14.75.62:43701)
0810 23:06:43.695798 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:43.695803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:43.696671 (+ 868us) server_negotiation.cc:407] Connection header received
0810 23:06:43.698235 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:43.698239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:43.698302 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:43.698384 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:43.699588 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.700156 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.701349 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.701676 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.701798 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:43.704865 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:43.704868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:43.704870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:43.704894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:43.706640 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.707216 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.707221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.707225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.707333 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.707673 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.707676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.707681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.707958 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:43.708076 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:43.708611 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 23:06:43.708771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:43.709848 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:43.696013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:60225)
0810 23:06:43.696506 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:43.696550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:43.697926 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:43.698556 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:43.698563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:43.698930 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:43.699422 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:43.699436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.700314 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.700317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:43.701211 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:43.701220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.701846 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.701850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:43.702029 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:43.702795 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:43.702813 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:43.704679 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:43.706781 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:43.706787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:43.706801 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:43.707095 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:43.707426 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:43.707428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:43.707429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:43.707574 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:43.708067 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:43.708073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:43.708352 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 23:06:43.708523 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 23:06:43.708817 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260810 23:06:43.711206 21256 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:43.711503 21256 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:43.712175 21256 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:43.713918 21062 ts_manager.cc:194] Registered new tserver with Master: ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961)
I20260810 23:06:43.715162 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:60225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
I20260810 23:06:43.732923 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:43.849289 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:43.849710 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:43.849799 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:43.854233 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:43.854342 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:43.854440 21261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:43.860114 21261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:43.861326 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:43.862430 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:43.872749 21261 server_base.cc:1034] running on GCE node
W20260810 23:06:43.872709 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:43.872699 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:43.872701 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:43.873519 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:43.874225 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:43.875416 21261 hybrid_clock.cc:648] HybridClock initialized: now 1786403203875401 us; error 32 us; skew 500 ppm
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:43.878405 21261 init.cc:377] Logged in from keytab as oryx/127.14.75.2@KRBTEST.COM (short username oryx)
I20260810 23:06:43.879348 21261 webserver.cc:460] Webserver started at http://127.14.75.2:34289/ using document root <none> and password file <none>
I20260810 23:06:43.879917 21261 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:43.879987 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:43.880169 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:43.882373 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "5b798b8faa177c94f8d5c4e9c3968d17"
server_key_iv: "6d2e2f602d4cd4325f0e68b9286addd9"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.882982 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "5b798b8faa177c94f8d5c4e9c3968d17"
server_key_iv: "6d2e2f602d4cd4325f0e68b9286addd9"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.887357 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:06:43.890000 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.891348 21261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:43.891490 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "5b798b8faa177c94f8d5c4e9c3968d17"
server_key_iv: "6d2e2f602d4cd4325f0e68b9286addd9"
server_key_version: "encryptionkey@0"
I20260810 23:06:43.891616 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:43.905668 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:43.909227 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:43.909507 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:43.910250 21261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:43.911157 21261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:43.911216 21261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.911291 21261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:43.911326 21261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:43.920920 21261 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36827
I20260810 23:06:43.920945 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36827 every 8 connection(s)
I20260810 23:06:43.921927 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:43.930471 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21261
I20260810 23:06:43.930635 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:43.931005 14636 external_mini_cluster.cc:1428] Setting key 7153a1a5803d56bed2ffeec3e9bca73d
Aug 10 23:06:43 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:43.937036 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:43.923936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:45925 (local address 127.14.75.62:43701)
0810 23:06:43.924131 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:43.924137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:43.925048 (+ 911us) server_negotiation.cc:407] Connection header received
0810 23:06:43.926472 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:43.926475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:43.926527 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:43.926609 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:43.927847 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.928408 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.929629 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.929852 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.929937 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:43.932762 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:43.932765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:43.932768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:43.932799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:43.934615 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.935329 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.935331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.935332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.935409 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:43.935830 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:43.935832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:43.935833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:43.936017 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:43.936096 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:43.936787 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 23:06:43.936903 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:06:43.938460 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:43.924372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:45925)
0810 23:06:43.924880 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:43.924926 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:43.926260 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:43.926784 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:43.926791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:43.927208 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:43.927710 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:43.927723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.928532 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.928534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:43.929519 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:43.929529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:43.929993 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:43.929995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:43.930156 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:43.930869 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:43.930888 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:43.932592 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:43.934780 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:43.934789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:43.934809 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:43.935227 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:43.935503 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:43.935509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:43.935512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:43.935724 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:43.936128 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:43.936138 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:43.936517 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 23:06:43.936751 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 23:06:43.937155 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260810 23:06:43.939743 21392 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:43.940064 21392 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:43.940605 21392 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
I20260810 23:06:43.941731 21062 ts_manager.cc:194] Registered new tserver with Master: 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
WARNING: no policy specified for oryx/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:43.942375 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:45925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
I20260810 23:06:43.969827 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:44.092612 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:44.092907 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:44.092952 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:44.097285 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:44.097353 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:44.097462 21397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:44.102958 21397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:44.104081 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:44.105077 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:44.115132 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.115135 21403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.115286 21404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.115538 21397 server_base.cc:1034] running on GCE node
I20260810 23:06:44.115998 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:44.116683 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:44.117873 21397 hybrid_clock.cc:648] HybridClock initialized: now 1786403204117836 us; error 51 us; skew 500 ppm
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:44.120757 21397 init.cc:377] Logged in from keytab as oryx/127.14.75.3@KRBTEST.COM (short username oryx)
I20260810 23:06:44.121817 21397 webserver.cc:460] Webserver started at http://127.14.75.3:34575/ using document root <none> and password file <none>
I20260810 23:06:44.122408 21397 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.122503 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.122753 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:44.124961 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "521a084e11734cb1b457b9c8f79c08fb"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "fc3e6a939e5d7d0871f9af73f7f52f36"
server_key_iv: "a1e6ed951808201f110ac183bdc76cfc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.125682 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "521a084e11734cb1b457b9c8f79c08fb"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "fc3e6a939e5d7d0871f9af73f7f52f36"
server_key_iv: "a1e6ed951808201f110ac183bdc76cfc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.129698 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:44.132267 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.133653 21397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:44.133797 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "521a084e11734cb1b457b9c8f79c08fb"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "fc3e6a939e5d7d0871f9af73f7f52f36"
server_key_iv: "a1e6ed951808201f110ac183bdc76cfc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.133921 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.144975 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:44.148059 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:44.148289 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:44.148909 21397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:44.149825 21397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:44.149899 21397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.149971 21397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:44.150015 21397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.159458 21397 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:33423
I20260810 23:06:44.159479 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:33423 every 8 connection(s)
I20260810 23:06:44.160421 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:44.168246 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21397
I20260810 23:06:44.168402 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:44.168694 14636 external_mini_cluster.cc:1428] Setting key d61440b9b47757225bd38559dddf051c
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:44.174795 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.162288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:52399 (local address 127.14.75.62:43701)
0810 23:06:44.162442 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.162445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.163295 (+ 850us) server_negotiation.cc:407] Connection header received
0810 23:06:44.164726 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.164730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.164800 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.164898 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.166175 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.166724 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.168002 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.168240 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.168353 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.171290 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.171293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.171295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.171319 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.172933 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.173400 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.173402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.173403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.173534 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.173855 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.173858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.173859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.174033 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.174109 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.174568 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.174675 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:06:44.175689 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.162667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:52399)
0810 23:06:44.163133 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.163179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.164482 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.165072 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.165080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.165536 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.166024 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.166038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.166867 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.166870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.167815 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.167825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.168402 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.168406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.168578 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.169384 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.169401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.171154 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.173036 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.173040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.173052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.173312 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.173617 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.173620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.173621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.173773 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.174120 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.174125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.174365 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.174521 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.174797 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260810 23:06:44.176899 21528 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:44.177146 21528 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:44.177718 21528 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
I20260810 23:06:44.178782 21062 ts_manager.cc:194] Registered new tserver with Master: 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423)
I20260810 23:06:44.179361 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:52399
I20260810 23:06:44.182760 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.61.keytab.
Entry for principal oryx/127.14.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.61.keytab.
I20260810 23:06:44.210479 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:36221
--webserver_interface=127.14.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.61.keytab
--principal=oryx/127.14.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:43701,127.14.75.61:36221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:44.326934 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:44.327244 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:44.327288 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:44.331597 21533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:44.331674 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:44.331696 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:44.331712 21533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:44.331728 21533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:44.337620 21533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:43701,127.14.75.61:36221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.61.keytab
--principal=oryx/127.14.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:36221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
W20260810 23:06:44.338068 21533 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.75.62:43701,127.14.75.61:36221'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:06:44.339136 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:44.340240 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:44.349479 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.349485 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.349485 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.349819 21533 server_base.cc:1034] running on GCE node
I20260810 23:06:44.350433 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:44.351574 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:44.352777 21533 hybrid_clock.cc:648] HybridClock initialized: now 1786403204352768 us; error 32 us; skew 500 ppm
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:44.355623 21533 init.cc:377] Logged in from keytab as oryx/127.14.75.61@KRBTEST.COM (short username oryx)
I20260810 23:06:44.357019 21533 webserver.cc:460] Webserver started at http://127.14.75.61:41897/ using document root <none> and password file <none>
I20260810 23:06:44.357651 21533 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.357754 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.358000 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:44.360147 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/instance:
uuid: "491c7b699dda43b880c20e2509d2be6a"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "f52e840f7f15c1b0638a397b163399a3"
server_key_iv: "f1553e2ea4a8d42e628374d541c7b95c"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.360809 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal/instance:
uuid: "491c7b699dda43b880c20e2509d2be6a"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "f52e840f7f15c1b0638a397b163399a3"
server_key_iv: "f1553e2ea4a8d42e628374d541c7b95c"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.364733 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:44.367478 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.369021 21533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:44.369160 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "491c7b699dda43b880c20e2509d2be6a"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "f52e840f7f15c1b0638a397b163399a3"
server_key_iv: "f1553e2ea4a8d42e628374d541c7b95c"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.369282 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.397492 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:44.400815 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:44.401049 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:44.408515 21533 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.61:36221
I20260810 23:06:44.408820 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.61:36221 every 8 connection(s)
I20260810 23:06:44.409749 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
I20260810 23:06:44.412768 21603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:44.416648 21603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:44.417900 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21533
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.61@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
I20260810 23:06:44.432058 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.418752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.14.75.62:43701)
0810 23:06:44.419027 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.419033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.419937 (+ 904us) server_negotiation.cc:407] Connection header received
0810 23:06:44.421015 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.421021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.421085 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.421188 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.422345 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.422881 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.424328 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.424654 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.424797 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.427661 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.427665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.427667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.427693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.429459 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.430165 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.430167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.430169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.430294 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.430632 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.430636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.430637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.430967 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.431106 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.431714 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.431862 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260810 23:06:44.433079 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.420181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.14.75.61:36221)
0810 23:06:44.420478 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.420489 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.420530 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:06:44.420796 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.420821 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.421042 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.422246 (+ 1204us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.422375 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.423686 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.424728 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.425212 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.425333 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.428085 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.428091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.428102 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.428123 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.430053 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.430754 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.430759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.430760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.430828 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.431195 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.431198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.431199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.431476 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.431700 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.431909 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.432082 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":184,"spinlock_wait_cycles":1280,"thread_start_us":90,"threads_started":1}
I20260810 23:06:44.433079 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.419256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47010)
0810 23:06:44.419769 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.419823 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.420786 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.421241 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.421250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.421687 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.422185 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.422200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.423021 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.423024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.424192 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.424204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.424835 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.424838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.424985 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.425693 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.425736 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.427470 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.429579 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.429588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.429598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.430045 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.430389 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.430391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.430393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.430527 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.431102 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.431108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.431465 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.431645 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.432027 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":158,"thread_start_us":145,"threads_started":1}
I20260810 23:06:44.435254 21603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.61@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
I20260810 23:06:44.443920 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.435796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42460)
0810 23:06:44.435933 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.435949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.436039 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.436262 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.436265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.436451 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.436538 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.436544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.437280 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.437285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.438003 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.438017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.438502 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.438505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.438602 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.439215 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.439227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.440707 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.442422 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.442429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.442434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.442697 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.442980 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.442983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.442985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.443146 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.443494 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.443496 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.443577 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.443670 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.443790 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:06:44.444043 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.435849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.14.75.61:36221)
0810 23:06:44.435933 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.435940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.435995 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:06:44.436124 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.436127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.436163 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.436252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.436656 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.437174 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.438132 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.438381 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.438496 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.440843 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.440846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.440848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.440870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.442301 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.442793 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.442795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.442796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.442873 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.443238 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.443240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.443241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.443414 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.443488 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.443716 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.443862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":25}
I20260810 23:06:44.447481 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Bootstrap starting.
I20260810 23:06:44.449903 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:44.450667 21603 log.cc:824] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:44.452785 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: No bootstrap required, opened a new log
I20260810 23:06:44.455962 21603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } }
I20260810 23:06:44.456187 21603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:44.456264 21603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491c7b699dda43b880c20e2509d2be6a, State: Initialized, Role: FOLLOWER
I20260810 23:06:44.456696 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } }
I20260810 23:06:44.457190 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } } }
I20260810 23:06:44.457310 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:44.458021 21603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:06:44.462882 21620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:44.463030 21620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:44.464674 21533 master_runner.cc:428] Detected that this master 491c7b699dda43b880c20e2509d2be6a is joining an existing cluster
I20260810 23:06:44.464732 21533 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.75.61:36221
I20260810 23:06:44.477206 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.467398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.14.75.62:43701)
0810 23:06:44.467524 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.467528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.467552 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:06:44.467631 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.467634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.467684 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.467781 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.468695 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.469514 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.470936 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.471302 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.471424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.473136 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.473140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.473143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.473173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.475410 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.475848 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.475851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.475853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.475942 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.476283 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.476286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.476288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.476475 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.476602 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.476897 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.477024 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:44.477311 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.467062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42476)
0810 23:06:44.467644 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.467668 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.467778 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.468016 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.468019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.468278 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.468448 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.468463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.469617 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.469621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.470793 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.470805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.471332 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.471334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.471438 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.472687 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.472704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.473172 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.475277 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.475281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.475283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.475523 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.476061 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.476063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.476065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.476164 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.476610 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.476612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.476730 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.476958 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.477100 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":197,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260810 23:06:44.477324 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.467434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.14.75.61:36221)
0810 23:06:44.467538 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.467542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.467758 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:06:44.467856 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.467859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.467906 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.468003 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.468633 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.469487 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.470921 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.471148 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.471203 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.473291 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.473294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.473296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.473317 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.475145 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.475621 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.475625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.475627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.475731 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.476258 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.476262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.476264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.476484 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.476577 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.476900 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.477158 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":40}
I20260810 23:06:44.477651 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.467001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47014)
0810 23:06:44.467292 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.467310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.467410 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.467909 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.467912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.468270 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.468472 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.468480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.469645 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.469650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.470794 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.470805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.471436 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.471439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.471573 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.472288 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.472305 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.472980 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.475523 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.475526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.475528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.475743 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.476037 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.476051 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.476053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.476159 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.476594 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.476597 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.476729 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.476841 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.477023 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":131,"spinlock_wait_cycles":10240,"thread_start_us":117,"threads_started":1}
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.62@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
I20260810 23:06:44.492321 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.482646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.14.75.61:36221)
0810 23:06:44.482768 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.482773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.483091 (+ 318us) server_negotiation.cc:407] Connection header received
0810 23:06:44.483301 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.483305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.483362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.483447 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.484110 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.484647 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.485716 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.486005 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.486114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.488932 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.488935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.488937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.488960 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.490470 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.490992 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.490997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.490999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.491083 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.491373 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.491377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.491379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.491572 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.491685 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.492034 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.492155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:06:44.492365 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.482609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42488)
0810 23:06:44.482967 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.483014 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.483167 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.483483 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.483489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.483857 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.483967 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.483973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.484762 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.484766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.485598 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.485608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.486127 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.486131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.486252 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.486796 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.486812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.488771 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.490610 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.490622 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.490628 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.490883 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.491180 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.491183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.491185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.491279 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.491685 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.491696 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.491829 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.491982 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.492205 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20260810 23:06:44.493847 21062 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.75.61:36221: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "491c7b699dda43b880c20e2509d2be6a"
member_type: NON_VOTER
last_known_addr {
host: "127.14.75.61"
port: 36221
}
attrs {
promote: true
}
}
dest_uuid: "cb837fb6159943aea1dfbdac2f5236d8"
cas_config_opid_index: -1
I20260810 23:06:44.494522 21062 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } }
I20260810 23:06:44.496727 21631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 491c7b699dda43b880c20e2509d2be6a (127.14.75.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } }
I20260810 23:06:44.497447 21576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:44.498716 21631 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.75.61:36221
I20260810 23:06:44.498915 21576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:06:44.499159 21533 master.cc:537] Master@127.14.75.61:36221 shutting down...
I20260810 23:06:44.499691 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.14.75.60@KRBTEST.COM; defaulting to no policy
I20260810 23:06:44.499837 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:44.501061 21632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.60@KRBTEST.COM" created.
W20260810 23:06:44.504344 21045 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 23:06:44.504431 21533 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:44.504746 21533 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:44.504892 21533 tablet_replica.cc:330] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: stopping tablet replica
W20260810 23:06:44.506067 21045 consensus_peers.cc:487] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 -> Peer 491c7b699dda43b880c20e2509d2be6a (127.14.75.61:36221): Couldn't send request to peer 491c7b699dda43b880c20e2509d2be6a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:06:44.520416 21533 master.cc:552] Master@127.14.75.61:36221 shutdown complete.
I20260810 23:06:44.520501 21533 master_runner.cc:305] Clearing existing system tablet
I20260810 23:06:44.520750 21533 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.520844 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.522327 21533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:06:44.523339 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.523561 21533 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.523693 21533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:44.523743 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "491c7b699dda43b880c20e2509d2be6a"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "f52e840f7f15c1b0638a397b163399a3"
server_key_iv: "f1553e2ea4a8d42e628374d541c7b95c"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.60.keytab.
Entry for principal oryx/127.14.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.60.keytab.
I20260810 23:06:44.525272 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:44343
--webserver_interface=127.14.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.60.keytab
--principal=oryx/127.14.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:44.526141 21533 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:44.529439 21533 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:06:44.529553 21533 log.cc:1196] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:06:44.529848 21533 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Deleting consensus metadata
I20260810 23:06:44.530041 21533 master_runner.cc:315] Copying system tablet from 127.14.75.62:43701
I20260810 23:06:44.531517 21533 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.75.62:43701
I20260810 23:06:44.541920 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.532172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47028)
0810 23:06:44.532488 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.532503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.532623 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.532963 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.532966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.533189 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.533326 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.533334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.534573 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.534576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.535693 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.535705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.536402 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.536406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.536504 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.537175 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.537190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.537691 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.540141 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.540145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.540147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.540413 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.540726 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.540729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.540731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.540826 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.541254 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.541257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.541364 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.541526 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.541698 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260810 23:06:44.541927 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.532321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.14.75.62:43701)
0810 23:06:44.532482 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.532489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.532598 (+ 109us) server_negotiation.cc:407] Connection header received
0810 23:06:44.532757 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.532762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.532827 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.532927 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.533504 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.534431 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.535833 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.536258 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.536400 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.537907 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.537913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.537919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.537959 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.540010 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.540527 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.540531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.540533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.540624 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.540922 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.540925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.540927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.541149 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.541248 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.541599 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.541765 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:44.542995 21082 tablet_copy_service.cc:140] P cb837fb6159943aea1dfbdac2f5236d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 491c7b699dda43b880c20e2509d2be6a ({username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:47028)
I20260810 23:06:44.543195 21082 tablet_copy_service.cc:161] P cb837fb6159943aea1dfbdac2f5236d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 491c7b699dda43b880c20e2509d2be6a at {username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:47028: session id = 491c7b699dda43b880c20e2509d2be6a-00000000000000000000000000000000
I20260810 23:06:44.545055 21082 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:06:44.546968 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:06:44.549507 21533 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: tablet copy: Starting download of 0 data blocks...
I20260810 23:06:44.549743 21533 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: tablet copy: Starting download of 1 WAL segments...
I20260810 23:06:44.551782 21533 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:06:44.553773 21082 tablet_copy_service.cc:342] P cb837fb6159943aea1dfbdac2f5236d8: Request end of tablet copy session 491c7b699dda43b880c20e2509d2be6a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:47028
I20260810 23:06:44.553925 21082 tablet_copy_service.cc:433] P cb837fb6159943aea1dfbdac2f5236d8: ending tablet copy session 491c7b699dda43b880c20e2509d2be6a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 491c7b699dda43b880c20e2509d2be6a
I20260810 23:06:44.561928 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:44.563875 21645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.564004 21648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.564169 21646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.569105 21533 server_base.cc:1034] running on GCE node
I20260810 23:06:44.569317 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:44.569797 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:44.571017 21533 hybrid_clock.cc:648] HybridClock initialized: now 1786403204571023 us; error 59 us; skew 500 ppm
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:44.572851 21533 init.cc:377] Logged in from keytab as oryx/127.14.75.61@KRBTEST.COM (short username oryx)
I20260810 23:06:44.573401 21533 webserver.cc:460] Webserver started at http://127.14.75.61:36089/ using document root <none> and password file <none>
I20260810 23:06:44.573694 21533 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.573743 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.575194 21533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:44.576264 21655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.576558 21533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:44.576647 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "491c7b699dda43b880c20e2509d2be6a"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "f52e840f7f15c1b0638a397b163399a3"
server_key_iv: "f1553e2ea4a8d42e628374d541c7b95c"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.576733 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.586954 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:44.589377 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:44.594808 21533 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.61:36221
I20260810 23:06:44.595055 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.61:36221 every 8 connection(s)
I20260810 23:06:44.595649 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
I20260810 23:06:44.597759 21709 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:06:44.598826 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Bootstrap starting.
I20260810 23:06:44.609614 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:44.610033 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Bootstrap complete.
I20260810 23:06:44.610680 21709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } }
I20260810 23:06:44.610829 21709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 491c7b699dda43b880c20e2509d2be6a, State: Initialized, Role: LEARNER
I20260810 23:06:44.610937 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } }
I20260810 23:06:44.611321 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } }
I20260810 23:06:44.611428 21709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:44.611541 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: LEARNER
I20260810 23:06:44.615259 21723 catalog_manager.cc:1259] Loaded cluster ID: 587399e21b03460cab9966f7f29d0af3
I20260810 23:06:44.615336 21723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: loading cluster ID for follower catalog manager: success
I20260810 23:06:44.617588 21723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: acquiring CA information for follower catalog manager: success
I20260810 23:06:44.619112 21723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:06:44.667248 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:44.667603 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:44.667691 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:44.672000 21638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:44.672107 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:44.672145 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:44.672190 21638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:44.672226 21638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:44.678053 21638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.60.keytab
--principal=oryx/127.14.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:44.679250 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:44.680320 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:44.689397 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.689397 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.689397 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.689458 21638 server_base.cc:1034] running on GCE node
I20260810 23:06:44.690256 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:44.691435 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:44.692627 21638 hybrid_clock.cc:648] HybridClock initialized: now 1786403204692613 us; error 28 us; skew 500 ppm
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:44.695976 21638 init.cc:377] Logged in from keytab as oryx/127.14.75.60@KRBTEST.COM (short username oryx)
I20260810 23:06:44.697016 21638 webserver.cc:460] Webserver started at http://127.14.75.60:36347/ using document root <none> and password file <none>
I20260810 23:06:44.697705 21638 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.697811 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.698050 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:44.700145 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/instance:
uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "9f52af678964bcb75abc941f6929ad22"
server_key_iv: "fd6b181af9eb50b6265f0b2ebe760cbc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.700747 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal/instance:
uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "9f52af678964bcb75abc941f6929ad22"
server_key_iv: "fd6b181af9eb50b6265f0b2ebe760cbc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.704842 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:44.707528 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.708833 21638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:06:44.708993 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "9f52af678964bcb75abc941f6929ad22"
server_key_iv: "fd6b181af9eb50b6265f0b2ebe760cbc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.709118 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.717896 21256 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
I20260810 23:06:44.722085 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:44.725631 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:44.725844 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:44.732880 21638 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.60:44343
I20260810 23:06:44.733175 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.60:44343 every 8 connection(s)
I20260810 23:06:44.734102 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
I20260810 23:06:44.737079 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:44.741623 21792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:44.744125 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21638
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.60@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
I20260810 23:06:44.756412 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.743741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.14.75.62:43701)
0810 23:06:44.743890 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.743896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.744814 (+ 918us) server_negotiation.cc:407] Connection header received
0810 23:06:44.746096 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.746099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.746147 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.746218 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.747185 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.747729 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.748837 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.749094 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.749174 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.752142 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.752149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.752155 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.752194 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.754018 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.754607 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.754611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.754613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.754711 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.755115 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.755119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.755120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.755403 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.755517 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.756059 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.756225 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:44.757261 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.744115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47044)
0810 23:06:44.744653 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.744709 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.745856 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.746294 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.746300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.746759 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.747036 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.747048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.747859 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.747862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.748713 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.748722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.749261 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.749267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.749527 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.750360 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.750376 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.751943 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.754133 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.754141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.754150 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.754505 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.754808 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.754812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.754817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.755005 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.755528 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.755533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.755799 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.755996 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.756357 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":78,"thread_start_us":190,"threads_started":1}
I20260810 23:06:44.757267 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.744906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.14.75.60:44343)
0810 23:06:44.745254 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.745265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.745306 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:06:44.745856 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.745879 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.746120 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.746315 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.747041 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.748054 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.748889 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.749318 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.749527 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.752614 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.752624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.752641 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.752670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.754596 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.755130 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.755136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.755137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.755219 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.755486 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.755489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.755491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.755791 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.755935 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.756220 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.756418 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":256,"thread_start_us":159,"threads_started":1}
I20260810 23:06:44.758594 21792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.60@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
I20260810 23:06:44.766932 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.759111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42500)
0810 23:06:44.759234 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.759248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.759353 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.759814 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.759816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.759959 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.760057 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.760062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.760936 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.760938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.761533 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.761542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.761972 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.761974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.762048 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.762505 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.762516 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.763746 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.765641 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.765643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.765645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.765839 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.766088 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.766090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.766091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.766164 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.766515 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.766517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.766602 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.766697 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.766805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:06:44.766992 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.759236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.14.75.61:36221)
0810 23:06:44.759613 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.759616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.759628 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:44.759665 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.759667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.759707 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.759787 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.760159 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.760842 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.761638 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.761876 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.761960 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.763879 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.763882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.763884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.763909 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.765539 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.765926 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.765928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.765929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.766007 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.766248 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.766250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.766251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.766427 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.766512 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.766755 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.766857 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
I20260810 23:06:44.767899 21792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.60@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
I20260810 23:06:44.775525 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.768416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45792)
0810 23:06:44.768524 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.768539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.768614 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.768893 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.768895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.769038 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.769109 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.769114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.769967 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.769970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.770453 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.770460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.770887 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.770889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.770944 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.771463 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.771474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.772675 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.774261 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.774263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.774265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.774453 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.774696 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.774698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.774699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.774772 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.775098 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.775100 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.775179 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.775259 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.775362 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 23:06:44.775527 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.768560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.14.75.60:44343)
0810 23:06:44.768713 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.768716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.768729 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:44.768759 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.768761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.768793 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.768876 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.769204 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.769865 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.770560 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.770792 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.770871 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.772799 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.772802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.772804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.772825 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.774158 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.774537 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.774539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.774540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.774610 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.774848 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.774850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.774851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.775019 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.775094 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.775316 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.775413 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:06:44.779222 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Bootstrap starting.
I20260810 23:06:44.781746 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:44.782534 21792 log.cc:824] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:44.784749 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: No bootstrap required, opened a new log
I20260810 23:06:44.787961 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } }
I20260810 23:06:44.788183 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:44.788285 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b54d7ba53e4fe2a76bf18d3ddd2c93, State: Initialized, Role: FOLLOWER
I20260810 23:06:44.788717 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } }
I20260810 23:06:44.789240 21798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } } }
I20260810 23:06:44.789377 21798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:44.790048 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:06:44.795303 21809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:44.795405 21809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:44.798754 21638 master_runner.cc:428] Detected that this master 42b54d7ba53e4fe2a76bf18d3ddd2c93 is joining an existing cluster
I20260810 23:06:44.798822 21638 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.75.60:44343
I20260810 23:06:44.810833 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.801497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.14.75.61:36221)
0810 23:06:44.801626 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.801631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.802315 (+ 684us) server_negotiation.cc:407] Connection header received
0810 23:06:44.802402 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.802406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.802462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.802537 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.803152 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.803898 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.805573 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.805891 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.806003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.807307 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.807311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.807318 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.807349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.809006 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.809444 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.809448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.809450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.809543 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.809877 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.809880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.809882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.810138 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.810234 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.810550 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.810675 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:06:44.810847 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.801022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42506)
0810 23:06:44.802204 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.802217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.802318 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.802658 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.802661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.802893 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.803014 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.803021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.804057 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.804060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.805437 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.805448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.806182 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.806188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.806287 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.806762 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.806774 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.807156 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.809119 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.809122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.809123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.809323 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.809671 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.809674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.809675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.809780 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.810284 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.810286 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.810369 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.810493 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.810649 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260810 23:06:44.811517 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21126
I20260810 23:06:44.823287 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:45961
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:43701
--webserver_port=38793
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:44.830474 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.801888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.14.75.60:44343)
0810 23:06:44.801987 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.801991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.802623 (+ 632us) server_negotiation.cc:407] Connection header received
0810 23:06:44.802761 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.802766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.802806 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.802850 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.803430 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.804201 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.805332 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.805678 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.806110 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.826744 (+ 20634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.826747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.826750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.826772 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.828765 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829224 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.829227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.829229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.829296 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829584 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.829587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.829589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.829805 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.829871 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.830168 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.830299 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:06:44.830814 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.800926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47054)
0810 23:06:44.801249 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.801269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.801373 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.801962 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.801966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.802254 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.802440 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.802449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.803543 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.803546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.804557 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.804568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.805301 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.805304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.805497 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.806091 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.806110 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.825966 (+ 19856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.828474 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.828479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.828484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.828920 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.829330 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.829339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.829479 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.830064 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.830071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.830227 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.830344 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.830536 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":78,"spinlock_wait_cycles":1408,"thread_start_us":132,"threads_started":1}
I20260810 23:06:44.831555 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.801892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45806)
0810 23:06:44.802548 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.802561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.802670 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.802964 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.802966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.803203 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.803316 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.803323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.804322 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.804325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.805209 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.805219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.806249 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.806252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.806342 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.826087 (+ 19745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.826120 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.826588 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.828883 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.828886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.828888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.829130 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.829383 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.829388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.829496 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.829926 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.829929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.830006 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.831211 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.831386 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":100,"threads_started":1}
I20260810 23:06:44.831609 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.801492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.14.75.62:43701)
0810 23:06:44.801612 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.801616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.801627 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:44.801663 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.801665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.801711 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.801807 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.802596 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.803405 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.804699 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.805039 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.805116 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.826227 (+ 21111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.826234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.826240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.826284 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.828297 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829072 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.829078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.829083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.829191 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.829606 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.829612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.829616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.829894 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.830008 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.831192 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.831379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403203, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.62@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
I20260810 23:06:44.846465 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.836223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45816)
0810 23:06:44.836423 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.836441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.836535 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.836832 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.836837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.837100 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.837303 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.837312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.838494 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.838498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.839452 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.839464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.840060 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.840064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.840167 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.840765 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.840781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.842517 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.844642 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.844645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.844648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.844950 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.845281 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.845286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.845289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.845391 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.845891 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.845894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.846005 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.846115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.846278 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 23:06:44.846487 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.836442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.14.75.60:44343)
0810 23:06:44.836547 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.836551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.836565 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:44.836644 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.836647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.836698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.836782 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.837486 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.838372 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.839587 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.839946 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.840055 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.842681 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.842685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.842688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.842717 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.844516 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.845067 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.845070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.845072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.845167 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.845515 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.845519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.845520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.845783 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.845886 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.846193 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.846328 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:06:44.847784 21062 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.75.60:44343: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
member_type: NON_VOTER
last_known_addr {
host: "127.14.75.60"
port: 44343
}
attrs {
promote: true
}
}
dest_uuid: "cb837fb6159943aea1dfbdac2f5236d8"
cas_config_opid_index: 5
I20260810 23:06:44.848459 21062 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } }
I20260810 23:06:44.850121 21766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:44.850540 21632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.851753 21766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:06:44.852213 21632 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.75.60:44343
I20260810 23:06:44.852403 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.852524 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:44.852649 21632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:44.852895 21638 master.cc:537] Master@127.14.75.60:44343 shutting down...
W20260810 23:06:44.854565 21047 consensus_peers.cc:487] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 -> Peer 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60:44343): Couldn't send request to peer 42b54d7ba53e4fe2a76bf18d3ddd2c93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:06:44.855870 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.849713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42520)
0810 23:06:44.849833 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.849847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.849928 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.850388 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.850391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.850409 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:44.850505 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.850511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.851719 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.851724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.853933 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.853945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.855363 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.855369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.855481 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.855544 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.855628 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.855743 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:06:44.855870 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.849970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.14.75.61:36221)
0810 23:06:44.850088 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.850108 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.850125 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:44.850180 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.850182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.850256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.850365 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:44.850658 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.851441 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.854094 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.854891 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.854979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.855160 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.855649 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.855741 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:44.856679 21682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 LEARNER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:06:44.857120 21631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:06:44.859014 21638 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:44.859357 21638 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:44.859157 21682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.859658 21638 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: stopping tablet replica
I20260810 23:06:44.861088 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb837fb6159943aea1dfbdac2f5236d8. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.861225 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: LEARNER
I20260810 23:06:44.861310 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: NON_VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: true } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.861382 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: LEARNER
I20260810 23:06:44.863119 21631 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: attempting to promote NON_VOTER 491c7b699dda43b880c20e2509d2be6a to VOTER
I20260810 23:06:44.863602 21631 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } }
I20260810 23:06:44.865473 21682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 LEARNER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:06:44.865891 21632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:06:44.866061 21638 master.cc:552] Master@127.14.75.60:44343 shutdown complete.
I20260810 23:06:44.866236 21638 master_runner.cc:305] Clearing existing system tablet
I20260810 23:06:44.866544 21638 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.865901 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.864913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45830)
0810 23:06:44.865719 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.865807 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696}
I20260810 23:06:44.867631 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.867222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45844)
0810 23:06:44.867498 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.867548 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:44343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260810 23:06:44.867820 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.868088 21682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 491c7b699dda43b880c20e2509d2be6a (127.14.75.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.867682 21819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 491c7b699dda43b880c20e2509d2be6a (127.14.75.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
W20260810 23:06:44.868465 21047 consensus_peers.cc:487] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 -> Peer 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60:44343): Couldn't send request to peer 42b54d7ba53e4fe2a76bf18d3ddd2c93. Status: Network error: Client connection negotiation failed: client connection to 127.14.75.60:44343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:06:44.869793 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.869922 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:44.869843 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.870102 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:44.870070 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.870151 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:44.873862 21638 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260810 23:06:44.874912 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.875162 21638 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.875298 21638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:44.875332 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "9f52af678964bcb75abc941f6929ad22"
server_key_iv: "fd6b181af9eb50b6265f0b2ebe760cbc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.877529 21638 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:44.880188 21638 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:06:44.880297 21638 log.cc:1196] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:06:44.880573 21638 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Deleting consensus metadata
I20260810 23:06:44.880762 21638 master_runner.cc:315] Copying system tablet from 127.14.75.62:43701
I20260810 23:06:44.882009 21638 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: tablet copy: Beginning tablet copy session from remote peer at address 127.14.75.62:43701
I20260810 23:06:44.892336 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.882714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.14.75.62:43701)
0810 23:06:44.882848 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:44.882853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:44.883046 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:06:44.883179 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:44.883182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:44.883236 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:44.883337 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:44.883927 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.884860 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.886248 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.886578 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.886686 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.888080 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:44.888084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:44.888087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:44.888124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:44.889939 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.890701 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.890705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.890706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.890846 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:44.891254 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:44.891257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:44.891259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:44.891499 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:44.891604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:44.892020 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 23:06:44.892168 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:44.892688 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:44.882594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:47056)
0810 23:06:44.882939 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:44.882953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:44.883079 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:44.883385 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:44.883388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:44.883630 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:44.883784 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.883793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.884969 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.884972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.886104 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:44.886116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:44.886715 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:44.886718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:44.886813 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:44.887430 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:44.887444 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:44.887924 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:44.890209 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.890215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.890219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.890552 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.890951 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:44.890956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:44.890960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:44.891102 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:44.891623 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:44.891628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:44.891828 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 23:06:44.891961 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:06:44.892132 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260810 23:06:44.893144 21082 tablet_copy_service.cc:140] P cb837fb6159943aea1dfbdac2f5236d8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 42b54d7ba53e4fe2a76bf18d3ddd2c93 ({username='oryx', principal='oryx/127.14.75.60@KRBTEST.COM'} at 127.0.0.1:47056)
I20260810 23:06:44.893253 21082 tablet_copy_service.cc:161] P cb837fb6159943aea1dfbdac2f5236d8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 42b54d7ba53e4fe2a76bf18d3ddd2c93 at {username='oryx', principal='oryx/127.14.75.60@KRBTEST.COM'} at 127.0.0.1:47056: session id = 42b54d7ba53e4fe2a76bf18d3ddd2c93-00000000000000000000000000000000
I20260810 23:06:44.894433 21082 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:06:44.895905 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:06:44.898540 21638 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: tablet copy: Starting download of 0 data blocks...
I20260810 23:06:44.898763 21638 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: tablet copy: Starting download of 1 WAL segments...
I20260810 23:06:44.901144 21638 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:06:44.903481 21082 tablet_copy_service.cc:342] P cb837fb6159943aea1dfbdac2f5236d8: Request end of tablet copy session 42b54d7ba53e4fe2a76bf18d3ddd2c93-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.75.60@KRBTEST.COM'} at 127.0.0.1:47056
I20260810 23:06:44.903590 21082 tablet_copy_service.cc:433] P cb837fb6159943aea1dfbdac2f5236d8: ending tablet copy session 42b54d7ba53e4fe2a76bf18d3ddd2c93-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 42b54d7ba53e4fe2a76bf18d3ddd2c93
I20260810 23:06:44.911568 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:44.913686 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:44.913692 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.914287 21638 server_base.cc:1034] running on GCE node
W20260810 23:06:44.917941 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:44.918236 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:44.918617 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:44.919787 21638 hybrid_clock.cc:648] HybridClock initialized: now 1786403204919763 us; error 47 us; skew 500 ppm
Aug 10 23:06:44 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403204, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:44.921761 21638 init.cc:377] Logged in from keytab as oryx/127.14.75.60@KRBTEST.COM (short username oryx)
I20260810 23:06:44.922250 21638 webserver.cc:460] Webserver started at http://127.14.75.60:36869/ using document root <none> and password file <none>
I20260810 23:06:44.922436 21638 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:44.922470 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:44.923960 21638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:44.925558 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:44.925933 21638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:44.926050 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "9f52af678964bcb75abc941f6929ad22"
server_key_iv: "fd6b181af9eb50b6265f0b2ebe760cbc"
server_key_version: "encryptionkey@0"
I20260810 23:06:44.926151 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:44.939986 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:44.942529 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:44.944857 21392 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
I20260810 23:06:44.947906 21638 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.60:44343
I20260810 23:06:44.947942 21895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.60:44343 every 8 connection(s)
I20260810 23:06:44.948557 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
I20260810 23:06:44.950506 21896 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:06:44.951624 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Bootstrap starting.
I20260810 23:06:44.964059 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:44.964427 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: Bootstrap complete.
I20260810 23:06:44.965097 21896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } }
I20260810 23:06:44.965253 21896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42b54d7ba53e4fe2a76bf18d3ddd2c93, State: Initialized, Role: LEARNER
I20260810 23:06:44.965363 21896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } }
I20260810 23:06:44.965775 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:44.965909 21896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:44.965991 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:06:44.969113 21910 catalog_manager.cc:1259] Loaded cluster ID: 587399e21b03460cab9966f7f29d0af3
I20260810 23:06:44.969255 21910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: loading cluster ID for follower catalog manager: success
I20260810 23:06:44.971701 21910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: acquiring CA information for follower catalog manager: success
I20260810 23:06:44.973067 21910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:06:44.984119 21818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:44.984498 21818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:44.984613 21818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:44.989094 21818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:44.989210 21818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:44.989310 21818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:44.996213 21818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:45961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=38793
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:44.997452 21818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:44.998684 21818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:45.008750 21915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.008919 21918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.008980 21916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:45.009193 21818 server_base.cc:1034] running on GCE node
I20260810 23:06:45.009744 21818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:45.010500 21818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:45.011742 21818 hybrid_clock.cc:648] HybridClock initialized: now 1786403205011723 us; error 30 us; skew 500 ppm
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:45.015012 21818 init.cc:377] Logged in from keytab as oryx/127.14.75.1@KRBTEST.COM (short username oryx)
I20260810 23:06:45.015993 21818 webserver.cc:460] Webserver started at http://127.14.75.1:38793/ using document root <none> and password file <none>
I20260810 23:06:45.016620 21818 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:45.016737 21818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:45.021144 21818 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:45.023519 21925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.024866 21818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:45.025029 21818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "ceb95e4ce6f8428d95cc08d447637af2"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "2151bb1c237a3849417bedbf15258b1b"
server_key_iv: "783033919cdb9b980dc19b49c80df941"
server_key_version: "encryptionkey@0"
I20260810 23:06:45.025753 21818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:45.048617 21818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:45.051918 21818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:45.052141 21818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:45.052781 21818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:45.053776 21818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:45.053848 21818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.053922 21818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:45.053966 21818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.063508 21818 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:45961
I20260810 23:06:45.063532 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:45961 every 8 connection(s)
I20260810 23:06:45.064913 21818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:45.073323 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 21818
I20260810 23:06:45.073566 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21261
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:45.082134 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.067165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:58505 (local address 127.14.75.61:36221)
0810 23:06:45.067293 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.067298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.068692 (+ 1394us) server_negotiation.cc:407] Connection header received
0810 23:06:45.069546 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.069549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.069642 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.069727 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.071258 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.072078 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073538 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.073886 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073967 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.077950 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.077955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.077959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.077999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.079875 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.080416 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.080420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.080422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.080512 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.080881 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.080885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.080887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.081093 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.081210 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.081833 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.081967 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:06:45.083007 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.067428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.14.75.1:58505)
0810 23:06:45.068576 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.068590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.069201 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.069815 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.069823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.070435 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.071089 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.071098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.072198 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.072201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.073365 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.073376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.074621 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.074626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.074861 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.075619 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.075643 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.077736 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.079998 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.080004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.080024 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.080300 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.080610 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.080614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.080616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.080775 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.081210 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.081219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.081566 (+ 347us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.081796 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.082145 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":204,"spinlock_wait_cycles":83072,"thread_start_us":205,"threads_started":1}
I20260810 23:06:45.085325 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.067161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:57145 (local address 127.14.75.62:43701)
0810 23:06:45.067293 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.067298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.068174 (+ 876us) server_negotiation.cc:407] Connection header received
0810 23:06:45.069454 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.069459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.069524 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.069594 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.071235 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.071780 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073538 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.073892 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073967 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.080033 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.080040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.080046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.080086 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.083515 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.083953 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.083959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.083964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.084079 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.084378 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.084382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.084384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.084590 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.084710 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.085028 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.085183 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:06:45.085325 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.067428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:57145)
0810 23:06:45.068011 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.068065 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.069201 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.069817 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.069825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.070412 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.071081 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.071095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.071931 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.071934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.073365 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.073376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.076695 (+ 3319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.076699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.076824 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.077469 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.077485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.079837 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.083636 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.083639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.083640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.083841 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.084173 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.084175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.084176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.084250 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.084763 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.084765 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.084846 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.084963 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.085116 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":139,"spinlock_wait_cycles":16000,"thread_start_us":721,"threads_started":1}
I20260810 23:06:45.086309 22045 heartbeater.cc:347] Connected to a master server at 127.14.75.61:36221
I20260810 23:06:45.086570 22045 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.086956 22043 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:45.087059 22043 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.087138 22045 heartbeater.cc:511] Master 127.14.75.61:36221 requested a full tablet report, sending...
I20260810 23:06:45.087253 22043 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
I20260810 23:06:45.087421 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:36827
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:43701
--webserver_port=34289
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:45.088454 21062 ts_manager.cc:194] Re-registered known tserver with Master: ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961)
I20260810 23:06:45.088717 21672 ts_manager.cc:194] Registered new tserver with Master: ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961)
I20260810 23:06:45.089277 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:57145
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
I20260810 23:06:45.099618 22050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.067429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.14.75.1:52737)
0810 23:06:45.068820 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.068832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.069211 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.070051 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.070054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.070415 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.071089 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.071098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.072416 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.072420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.073207 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.073220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.089724 (+ 16504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.089729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.089841 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.093569 (+ 3728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.093582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.095236 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.097808 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.097811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.097813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.098039 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.098423 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.098425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.098426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.098533 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.099072 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.099098 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.099194 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.099283 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.099452 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"mutex_wait_us":269,"spinlock_wait_cycles":18560,"thread_start_us":151,"threads_started":1}
I20260810 23:06:45.099803 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.066909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:52737 (local address 127.14.75.60:44343)
0810 23:06:45.067437 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.067451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.068915 (+ 1464us) server_negotiation.cc:407] Connection header received
0810 23:06:45.069855 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.069859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.069912 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.070014 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.071258 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.072304 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073365 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.073824 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.073927 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.095454 (+ 21527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.095462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.095467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.095510 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.097661 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.098168 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.098172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.098174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.098288 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.098646 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.098652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.098655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.098948 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.099073 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.099391 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.099563 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":84,"threads_started":1}
I20260810 23:06:45.100808 22042 heartbeater.cc:347] Connected to a master server at 127.14.75.60:44343
I20260810 23:06:45.100914 22042 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.101183 22042 heartbeater.cc:511] Master 127.14.75.60:44343 requested a full tablet report, sending...
I20260810 23:06:45.102380 21860 ts_manager.cc:194] Registered new tserver with Master: ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961)
I20260810 23:06:45.181383 21528 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
W20260810 23:06:45.208528 22051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:45.208863 22051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:45.208942 22051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:45.213276 22051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:45.213451 22051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:45.213536 22051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:45.218955 22051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:36827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=34289
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:45.220063 22051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:45.221053 22051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:45.231226 22057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.231231 22059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.231271 22056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:45.231654 22051 server_base.cc:1034] running on GCE node
I20260810 23:06:45.232023 22051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:45.232673 22051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:45.233832 22051 hybrid_clock.cc:648] HybridClock initialized: now 1786403205233818 us; error 25 us; skew 500 ppm
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:45.236738 22051 init.cc:377] Logged in from keytab as oryx/127.14.75.2@KRBTEST.COM (short username oryx)
I20260810 23:06:45.237834 22051 webserver.cc:460] Webserver started at http://127.14.75.2:34289/ using document root <none> and password file <none>
I20260810 23:06:45.238569 22051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:45.238695 22051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:45.242817 22051 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:06:45.244932 22067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.246212 22051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:45.246359 22051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
format_stamp: "Formatted at 2026-08-10 23:06:43 on dist-test-slave-srgv"
server_key: "5b798b8faa177c94f8d5c4e9c3968d17"
server_key_iv: "6d2e2f602d4cd4325f0e68b9286addd9"
server_key_version: "encryptionkey@0"
I20260810 23:06:45.246930 22051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:45.263599 22051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:45.266948 22051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:45.267174 22051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:45.267854 22051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:45.268778 22051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:45.268834 22051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.268906 22051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:45.268939 22051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.278630 22051 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36827
I20260810 23:06:45.278663 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36827 every 8 connection(s)
I20260810 23:06:45.279681 22051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:45.286087 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22051
I20260810 23:06:45.286294 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21397
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
I20260810 23:06:45.298933 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:33423
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:43701
--webserver_port=34575
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:45.299779 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.282684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:56693 (local address 127.14.75.60:44343)
0810 23:06:45.282923 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.282927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.283386 (+ 459us) server_negotiation.cc:407] Connection header received
0810 23:06:45.284515 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.284518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.284571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.284626 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.286647 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.287486 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.289279 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.289691 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.289763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.294974 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.294980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.294994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.295031 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.296897 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297696 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.297699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.297701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.297796 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.298277 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.298281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.298283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.298512 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.298598 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.299440 (+ 842us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.299589 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260810 23:06:45.299830 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.282514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:47569 (local address 127.14.75.61:36221)
0810 23:06:45.282824 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.282829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.283863 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 23:06:45.284514 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.284518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.284571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.284626 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.287770 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.288523 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.289607 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.289939 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.290009 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.294749 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.294756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.294762 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.294799 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.296735 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297729 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.297736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.297741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.297840 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.298289 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.298295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.298300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.298565 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.298652 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.299438 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.299593 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260810 23:06:45.300549 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.281913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:55861 (local address 127.14.75.62:43701)
0810 23:06:45.282390 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.282395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.283453 (+ 1058us) server_negotiation.cc:407] Connection header received
0810 23:06:45.284494 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.284498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.284558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.284662 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.286643 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.287485 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.289461 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.289888 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.290008 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.294117 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.294121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.294124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.294159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.296980 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297697 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.297700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.297702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.297796 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.298271 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.298274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.298275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.298512 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.298598 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.300300 (+ 1702us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.300440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260810 23:06:45.301266 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.282199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.14.75.2:47569)
0810 23:06:45.283766 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.283776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.284289 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.284924 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.284931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.285587 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.287610 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.287622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.288666 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.288669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.289477 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.289486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.290157 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.290160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.290262 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.291274 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.291290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.294571 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.296884 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.296893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.296916 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.297602 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.297959 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.297963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.298155 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.298716 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.298722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.299126 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.299992 (+ 866us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.300200 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":373,"spinlock_wait_cycles":15616,"thread_start_us":135,"threads_started":1}
I20260810 23:06:45.301568 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.282199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:55861)
0810 23:06:45.283312 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.283324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.284288 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.284921 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.284929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.285560 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.286467 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.286484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.287668 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.287675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.289248 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.289264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.290180 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.290187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.290368 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.291101 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.291135 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.293918 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.297093 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.297102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.297560 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.297912 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.297917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.298148 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.298686 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.298688 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.300077 (+ 1389us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.300777 (+ 700us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.300924 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":1299,"spinlock_wait_cycles":88448,"thread_start_us":188,"threads_started":1}
I20260810 23:06:45.302464 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.282199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.14.75.2:56693)
0810 23:06:45.283187 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.283247 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.284289 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.284924 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.284931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.285587 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.286464 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.286484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.287656 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.287662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.289131 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.289140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.289973 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.289979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.290214 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.291182 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.291202 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.294755 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.297036 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.297046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.297580 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.297912 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.297915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.297917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.298148 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.298675 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.298681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.299127 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.299382 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.299745 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":403,"spinlock_wait_cycles":40064,"thread_start_us":706,"threads_started":1}
I20260810 23:06:45.302619 22186 heartbeater.cc:347] Connected to a master server at 127.14.75.61:36221
I20260810 23:06:45.302870 22186 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.303254 22184 heartbeater.cc:347] Connected to a master server at 127.14.75.60:44343
I20260810 23:06:45.303339 22184 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.303471 22186 heartbeater.cc:511] Master 127.14.75.61:36221 requested a full tablet report, sending...
I20260810 23:06:45.303490 22184 heartbeater.cc:511] Master 127.14.75.60:44343 requested a full tablet report, sending...
I20260810 23:06:45.304065 22185 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:45.304142 22185 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.304275 22185 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
I20260810 23:06:45.304821 21860 ts_manager.cc:194] Registered new tserver with Master: 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:45.304868 21062 ts_manager.cc:194] Re-registered known tserver with Master: 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:45.305155 21672 ts_manager.cc:194] Registered new tserver with Master: 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:45.305657 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:55861
W20260810 23:06:45.420789 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:45.421183 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:45.421247 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:06:45.422099 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.418266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:45850)
0810 23:06:45.418603 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.418615 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.418756 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.419080 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.419082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.419102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.419229 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.419235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.420008 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.420011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.420791 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.420800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.421685 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.421689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.421783 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.421809 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.421878 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.421960 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":81,"threads_started":1}
I20260810 23:06:45.422096 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.418333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.14.75.60:44343)
0810 23:06:45.418467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.418474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.418714 (+ 240us) server_negotiation.cc:407] Connection header received
0810 23:06:45.418880 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.418886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.418951 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.419063 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.419342 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.419880 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.420924 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.421542 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.421664 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.421855 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.421896 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.421961 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:45.423074 21865 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 23:06:45.423655 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb837fb6159943aea1dfbdac2f5236d8. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:45.423817 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: LEARNER
W20260810 23:06:45.426446 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:45.426553 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:45.426656 22191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:45.432404 22191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:33423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=34575
--tserver_master_addrs=127.14.75.62:43701,127.14.75.61:36221,127.14.75.60:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:45.433633 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:45.434688 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:45.444821 22199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.444845 22198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:45.444875 22201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:45.445355 22191 server_base.cc:1034] running on GCE node
I20260810 23:06:45.445798 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:45.446503 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:45.447681 22191 hybrid_clock.cc:648] HybridClock initialized: now 1786403205447665 us; error 27 us; skew 500 ppm
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:45.450764 22191 init.cc:377] Logged in from keytab as oryx/127.14.75.3@KRBTEST.COM (short username oryx)
I20260810 23:06:45.451793 22191 webserver.cc:460] Webserver started at http://127.14.75.3:34575/ using document root <none> and password file <none>
I20260810 23:06:45.452385 22191 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:45.452504 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:45.456542 22191 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:45.458841 22208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.460152 22191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:45.460300 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "521a084e11734cb1b457b9c8f79c08fb"
format_stamp: "Formatted at 2026-08-10 23:06:44 on dist-test-slave-srgv"
server_key: "fc3e6a939e5d7d0871f9af73f7f52f36"
server_key_iv: "a1e6ed951808201f110ac183bdc76cfc"
server_key_version: "encryptionkey@0"
I20260810 23:06:45.460888 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:45.475625 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:45.479318 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:45.479493 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:45.480060 22191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:45.480919 22191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:45.480969 22191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.481009 22191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:45.481029 22191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:45.490471 22191 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:33423
I20260810 23:06:45.490556 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:33423 every 8 connection(s)
I20260810 23:06:45.491582 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:45.497555 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22191
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.60@KRBTEST.COM
Aug 10 23:06:45 dist-test-slave-srgv krb5kdc[21016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403205, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.61@KRBTEST.COM
I20260810 23:06:45.510912 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.494126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.14.75.3:46555)
0810 23:06:45.494738 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.494804 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.495981 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.496552 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.496559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.497046 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.497853 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.497875 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.499050 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.499053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.500823 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.500832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.502071 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.502608 (+ 537us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.502722 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.503217 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.503232 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.505826 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.507988 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.507990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.507992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.508208 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.508781 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.508784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.508786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.509291 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.509797 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.509799 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.509883 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.509969 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.510099 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":89,"spinlock_wait_cycles":99328,"thread_start_us":153,"threads_started":1}
I20260810 23:06:45.511672 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.493564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:46555 (local address 127.14.75.60:44343)
0810 23:06:45.493806 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.493812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.494919 (+ 1107us) server_negotiation.cc:407] Connection header received
0810 23:06:45.496235 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.496239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.496298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.496355 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.498042 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.498878 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.501603 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.501924 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.502059 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.506008 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.506012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.506016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.506049 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.507878 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.508316 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.508320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.508321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.508668 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.509398 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.509401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.509431 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.509641 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.509752 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.511373 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.511514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 23:06:45.511735 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.494126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:43177)
0810 23:06:45.494738 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.494804 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.495981 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.496552 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.496559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.497046 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.497853 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.497875 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.499050 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.499053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.500693 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.500717 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.501550 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.501553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.501796 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.502527 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.502555 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.504801 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.507131 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.507139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.507161 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.507570 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.507915 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.507921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.507926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.508127 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.508732 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.508742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.509103 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.509332 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.509738 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":222,"spinlock_wait_cycles":77568,"thread_start_us":156,"threads_started":1}
I20260810 23:06:45.511885 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.493640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:43177 (local address 127.14.75.62:43701)
0810 23:06:45.493786 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.493792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.494921 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 23:06:45.496237 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.496241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.496298 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.496355 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.498042 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.498874 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.500849 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.501185 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.501299 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.505017 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.505021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.505024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.505053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.506984 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.507702 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.507706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.507711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.507805 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.508301 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.508304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.508306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.508550 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.508656 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.511545 (+ 2889us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.511714 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":4480}
I20260810 23:06:45.512867 22325 heartbeater.cc:347] Connected to a master server at 127.14.75.60:44343
I20260810 23:06:45.513099 22326 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43701
I20260810 23:06:45.513144 22325 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.513193 22326 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.513859 22325 heartbeater.cc:511] Master 127.14.75.60:44343 requested a full tablet report, sending...
I20260810 23:06:45.513859 22326 heartbeater.cc:511] Master 127.14.75.62:43701 requested a full tablet report, sending...
I20260810 23:06:45.515220 21860 ts_manager.cc:194] Registered new tserver with Master: 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423)
I20260810 23:06:45.515647 21062 ts_manager.cc:194] Re-registered known tserver with Master: 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423)
I20260810 23:06:45.516479 21062 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:43177
I20260810 23:06:45.516718 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.495464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:54635 (local address 127.14.75.61:36221)
0810 23:06:45.495597 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.495602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.495734 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:06:45.496208 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.496211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.496269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.496372 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.498051 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.498850 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.500851 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.501201 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.501286 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.512359 (+ 11073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.512363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.512365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.512398 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.514211 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.515335 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.515339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.515340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.515440 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.515786 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.515789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.515791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.516016 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.516093 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.516445 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.516564 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:06:45.517138 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.495333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.14.75.3:54635)
0810 23:06:45.495636 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.495648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.495989 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.496550 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.496557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.497040 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:45.497864 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.497875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.499042 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.499045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.500694 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.500708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.501550 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.501554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.502215 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.502641 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:45.502653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:45.512143 (+ 9490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:45.514899 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.514905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.514908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.515202 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.515556 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:45.515559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:45.515561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:45.515657 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:45.516167 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:45.516169 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:45.516263 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.516383 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.516552 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":274,"thread_start_us":124,"threads_started":1}
I20260810 23:06:45.517803 22328 heartbeater.cc:347] Connected to a master server at 127.14.75.61:36221
I20260810 23:06:45.517899 22328 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:45.518060 22328 heartbeater.cc:511] Master 127.14.75.61:36221 requested a full tablet report, sending...
I20260810 23:06:45.518783 21672 ts_manager.cc:194] Registered new tserver with Master: 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423)
I20260810 23:06:45.518857 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.501505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.14.75.61:36221)
0810 23:06:45.501765 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.501771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.504154 (+ 2383us) server_negotiation.cc:407] Connection header received
0810 23:06:45.504376 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.504380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.504441 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.504534 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.505603 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.506403 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.507485 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.507867 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.507959 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.514630 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.514634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.514636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.514669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.516414 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.516969 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.516975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.516979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.517072 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.517435 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.517441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.517443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.517659 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.517790 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.518523 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.518659 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260810 23:06:45.519629 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.499879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.14.75.60:44343)
0810 23:06:45.500484 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.500490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.500861 (+ 371us) server_negotiation.cc:407] Connection header received
0810 23:06:45.500903 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.500905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.500953 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.501025 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.502400 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.503237 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.506529 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.506892 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.506992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.510300 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.510305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.510308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.510337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.513459 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.514005 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.514008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.514010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.514100 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.514425 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.514427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.514429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.517557 (+ 3128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.517698 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.519314 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.519463 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":94,"threads_started":1}
I20260810 23:06:45.521041 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.499651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.14.75.62:43701)
0810 23:06:45.501590 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.501606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.504377 (+ 2771us) server_negotiation.cc:407] Connection header received
0810 23:06:45.504421 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.504425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.504481 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.505458 (+ 977us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.505583 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.506393 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.508108 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.508472 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.508606 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.511477 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.511485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.511493 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.511540 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.519426 (+ 7886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.519974 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.519978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.519982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.520075 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.520359 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.520361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.520362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.520541 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.520649 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.520823 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.520924 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1860,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260810 23:06:45.523670 21062 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:45.526767 21062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:45.529628 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: NON_VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: true } } }
I20260810 23:06:45.529816 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:45.533515 22192 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: attempting to promote NON_VOTER 42b54d7ba53e4fe2a76bf18d3ddd2c93 to VOTER
I20260810 23:06:45.533849 22192 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } }
I20260810 23:06:45.535151 21682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 23:06:45.535338 21865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 LEARNER]: Refusing update from remote peer cb837fb6159943aea1dfbdac2f5236d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 23:06:45.535625 22345 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 23:06:45.535802 22344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 23:06:45.537976 22344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.538602 21682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.539749 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.539896 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:45.539671 21865 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.540037 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.540172 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:45.539768 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.540530 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:45.541755 22353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.541989 22353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:45.541985 21899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb837fb6159943aea1dfbdac2f5236d8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:45.542125 21899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:45.550143 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.544511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:33423 (local address 127.0.0.1:55448)
0810 23:06:45.544710 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.544729 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.544825 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.545710 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.545717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.545736 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.545849 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.545855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.547178 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.547182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.548529 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.548538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.549753 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.549757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.549848 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.549877 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.549929 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.550000 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":32}
I20260810 23:06:45.550874 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.544688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.14.75.3:33423)
0810 23:06:45.545081 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.545096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.545121 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:06:45.545206 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.545213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.545521 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.546013 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.546134 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.547045 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.548669 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.549609 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.550354 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.550531 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.550604 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.550706 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":145,"threads_started":1}
I20260810 23:06:45.551113 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.544671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:45961 (local address 127.0.0.1:39106)
0810 23:06:45.545078 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.545108 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.545184 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.546433 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.546437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.546460 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.546557 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.546563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.548128 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.548131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.549266 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.549277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.550633 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.550637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.550726 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.550749 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.550821 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.550901 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":22,"thread_start_us":119,"threads_started":1}
I20260810 23:06:45.551548 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.545135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.14.75.1:45961)
0810 23:06:45.545477 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.545518 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.545543 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:06:45.545632 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.545639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.545847 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.546700 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.546819 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.547984 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.549451 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.550358 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.550455 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.550570 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.551296 (+ 726us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.551400 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260810 23:06:45.553908 21974 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 (DEFAULT_TABLE table=test-table [id=797490d8d72d4eddbea3588b9a729726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:45.554121 22258 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 (DEFAULT_TABLE table=test-table [id=797490d8d72d4eddbea3588b9a729726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:45.554845 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2e844f6ae42c0b6f8b0818b90cff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:45.555136 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.545597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36827 (local address 127.0.0.1:37196)
0810 23:06:45.548243 (+ 2646us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.548259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.548356 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.548928 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.549259 (+ 331us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.549286 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.549400 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.549408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.550855 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.550862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.552245 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.552257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.554673 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.554677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.554778 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.554805 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.554890 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.554989 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2574,"thread_start_us":106,"threads_started":1}
I20260810 23:06:45.555256 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2e844f6ae42c0b6f8b0818b90cff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:45.555624 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.546021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.14.75.2:36827)
0810 23:06:45.546368 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.546380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.548457 (+ 2077us) server_negotiation.cc:407] Connection header received
0810 23:06:45.548550 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.548557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.548777 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.548956 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.549589 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.550726 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.552408 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.554306 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.554425 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.554594 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.555339 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.555450 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260810 23:06:45.558632 22115 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 (DEFAULT_TABLE table=test-table [id=797490d8d72d4eddbea3588b9a729726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:45.559808 22115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2e844f6ae42c0b6f8b0818b90cff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:45.563181 22360 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Bootstrap starting.
I20260810 23:06:45.564646 22361 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Bootstrap starting.
I20260810 23:06:45.566805 22360 tablet_bootstrap.cc:654] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:45.567159 22361 tablet_bootstrap.cc:654] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:45.567873 22360 log.cc:824] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:45.567992 22361 log.cc:824] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:45.568159 22362 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Bootstrap starting.
I20260810 23:06:45.570283 22360 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: No bootstrap required, opened a new log
I20260810 23:06:45.570310 22361 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: No bootstrap required, opened a new log
I20260810 23:06:45.570516 22360 ts_tablet_manager.cc:1397] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:06:45.570533 22361 ts_tablet_manager.cc:1397] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:06:45.570544 22362 tablet_bootstrap.cc:654] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:45.571503 22362 log.cc:824] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:45.573911 22362 tablet_bootstrap.cc:492] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: No bootstrap required, opened a new log
I20260810 23:06:45.574195 22362 ts_tablet_manager.cc:1397] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:06:45.575114 22360 raft_consensus.cc:348] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.575194 22361 raft_consensus.cc:348] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.575361 22360 raft_consensus.cc:374] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:45.575459 22360 raft_consensus.cc:729] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521a084e11734cb1b457b9c8f79c08fb, State: Initialized, Role: FOLLOWER
I20260810 23:06:45.575464 22361 raft_consensus.cc:374] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:45.575522 22361 raft_consensus.cc:729] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb95e4ce6f8428d95cc08d447637af2, State: Initialized, Role: FOLLOWER
I20260810 23:06:45.575966 22360 consensus_queue.cc:260] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.575974 22361 consensus_queue.cc:260] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.577135 22361 ts_tablet_manager.cc:1428] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:06:45.577193 22360 ts_tablet_manager.cc:1428] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:06:45.577453 22326 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
I20260810 23:06:45.579587 22043 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
I20260810 23:06:45.583179 22362 raft_consensus.cc:348] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.583639 22362 raft_consensus.cc:374] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:45.583722 22362 raft_consensus.cc:729] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a9e4d53e43a996b53162d58f1c3c, State: Initialized, Role: FOLLOWER
I20260810 23:06:45.584245 22362 consensus_queue.cc:260] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.584997 22185 heartbeater.cc:503] Master 127.14.75.62:43701 was elected leader, sending a full tablet report...
I20260810 23:06:45.585361 22362 ts_tablet_manager.cc:1428] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260810 23:06:45.704596 22366 raft_consensus.cc:482] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:45.704887 22366 raft_consensus.cc:504] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.706115 22366 leader_election.cc:288] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423), 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:45.710196 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.706542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:45543 (local address 127.14.75.2:36827)
0810 23:06:45.706694 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.706699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.706862 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:06:45.706972 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.706975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.707031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.707115 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.707454 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.708000 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709176 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.709600 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.709764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.709982 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.710061 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:06:45.710227 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.706426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:33423 (local address 127.14.75.1:43725)
0810 23:06:45.706877 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.706893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.706998 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.707274 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.707276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.707291 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.707403 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.707411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.708125 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.708129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.709038 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.709047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.709794 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.709897 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.709914 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.710011 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.710087 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":36,"thread_start_us":86,"threads_started":1}
I20260810 23:06:45.710227 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.706560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:43725 (local address 127.14.75.3:33423)
0810 23:06:45.706689 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.706694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.707006 (+ 312us) server_negotiation.cc:407] Connection header received
0810 23:06:45.707108 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.707111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.707161 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.707241 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:45.707519 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.708004 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709179 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.709635 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709772 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.709837 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.710044 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.710122 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:45.710230 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.706404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36827 (local address 127.14.75.1:45543)
0810 23:06:45.706763 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:45.706777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:45.706880 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:45.707138 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:45.707141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:45.707159 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:45.707323 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.707331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.708122 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.708126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.709040 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:45.709050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.709722 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.709729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:45.709838 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.709871 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:06:45.709942 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:06:45.710037 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":63,"thread_start_us":104,"threads_started":1}
I20260810 23:06:45.711131 22136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2e844f6ae42c0b6f8b0818b90cff7" candidate_uuid: "ceb95e4ce6f8428d95cc08d447637af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" is_pre_election: true
I20260810 23:06:45.711112 22278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2e844f6ae42c0b6f8b0818b90cff7" candidate_uuid: "ceb95e4ce6f8428d95cc08d447637af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521a084e11734cb1b457b9c8f79c08fb" is_pre_election: true
I20260810 23:06:45.711462 22278 raft_consensus.cc:2454] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb95e4ce6f8428d95cc08d447637af2 in term 0.
I20260810 23:06:45.711462 22136 raft_consensus.cc:2454] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb95e4ce6f8428d95cc08d447637af2 in term 0.
I20260810 23:06:45.711957 21927 leader_election.cc:302] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 521a084e11734cb1b457b9c8f79c08fb, 72f3a9e4d53e43a996b53162d58f1c3c, ceb95e4ce6f8428d95cc08d447637af2; no voters:
I20260810 23:06:45.712303 22366 raft_consensus.cc:2781] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:45.712389 22366 raft_consensus.cc:482] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:45.712440 22366 raft_consensus.cc:3037] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:45.713649 22366 raft_consensus.cc:504] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.714038 22366 leader_election.cc:288] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [CANDIDATE]: Term 1 election: Requested vote from peers 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423), 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:45.714461 22136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2e844f6ae42c0b6f8b0818b90cff7" candidate_uuid: "ceb95e4ce6f8428d95cc08d447637af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
I20260810 23:06:45.714461 22278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2e844f6ae42c0b6f8b0818b90cff7" candidate_uuid: "ceb95e4ce6f8428d95cc08d447637af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521a084e11734cb1b457b9c8f79c08fb"
I20260810 23:06:45.714597 22278 raft_consensus.cc:3037] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:45.714600 22136 raft_consensus.cc:3037] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:45.716001 22136 raft_consensus.cc:2454] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb95e4ce6f8428d95cc08d447637af2 in term 1.
I20260810 23:06:45.716001 22278 raft_consensus.cc:2454] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb95e4ce6f8428d95cc08d447637af2 in term 1.
I20260810 23:06:45.716307 21927 leader_election.cc:302] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521a084e11734cb1b457b9c8f79c08fb, ceb95e4ce6f8428d95cc08d447637af2; no voters:
I20260810 23:06:45.716516 22366 raft_consensus.cc:2781] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:45.716763 22366 raft_consensus.cc:686] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 LEADER]: Becoming Leader. State: Replica: ceb95e4ce6f8428d95cc08d447637af2, State: Running, Role: LEADER
I20260810 23:06:45.717110 22366 consensus_queue.cc:237] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:45.720680 21060 catalog_manager.cc:5495] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "ceb95e4ce6f8428d95cc08d447637af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:45.741014 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.737321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.14.75.1:45961)
0810 23:06:45.737482 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.737488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.737519 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:06:45.737609 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.737611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.737670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.737784 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:45.738073 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.738626 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.739542 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.739766 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.739875 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.740202 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:45.740625 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:45.740763 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.740804 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.740860 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 23:06:45.742738 22329 tablet.cc:2369] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:45.747042 22278 raft_consensus.cc:1261] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Refusing update from remote peer ceb95e4ce6f8428d95cc08d447637af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:45.747043 22136 raft_consensus.cc:1261] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Refusing update from remote peer ceb95e4ce6f8428d95cc08d447637af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:45.747818 22371 consensus_queue.cc:1035] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:45.748015 22366 consensus_queue.cc:1035] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:45.756541 22374 mvcc.cc:205] Tried to move back new op lower bound from 7317107530732048384 to 7317107530620252160. Current Snapshot: MvccSnapshot[applied={T|T < 7317107530732048384}]
I20260810 23:06:45.762722 22376 mvcc.cc:205] Tried to move back new op lower bound from 7317107530732048384 to 7317107530620252160. Current Snapshot: MvccSnapshot[applied={T|T < 7317107530732048384 or (T in {7317107530732048384})}]
I20260810 23:06:45.764315 22375 mvcc.cc:205] Tried to move back new op lower bound from 7317107530732048384 to 7317107530620252160. Current Snapshot: MvccSnapshot[applied={T|T < 7317107530732048384}]
I20260810 23:06:45.766577 21062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:45.766733 21062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:45.771319 21062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb837fb6159943aea1dfbdac2f5236d8: Sending DeleteTablet for 3 replicas of tablet 08b2e844f6ae42c0b6f8b0818b90cff7
I20260810 23:06:45.772255 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:45 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39106
I20260810 23:06:45.772414 22115 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:45 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:37196
I20260810 23:06:45.773156 22384 tablet_replica.cc:330] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: stopping tablet replica
I20260810 23:06:45.773438 22258 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b2e844f6ae42c0b6f8b0818b90cff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:45 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:55448
I20260810 23:06:45.773696 22384 raft_consensus.cc:2229] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:45.774245 22384 raft_consensus.cc:2258] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:45.774698 22386 tablet_replica.cc:330] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: stopping tablet replica
I20260810 23:06:45.775256 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21029
I20260810 23:06:45.777660 22384 ts_tablet_manager.cc:1905] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:45.780728 22384 ts_tablet_manager.cc:1918] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:45.780910 22384 log.cc:1196] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/08b2e844f6ae42c0b6f8b0818b90cff7
I20260810 23:06:45.781253 22384 ts_tablet_manager.cc:1939] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting consensus metadata
I20260810 23:06:45.783046 22386 raft_consensus.cc:2229] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:45.783437 22386 raft_consensus.cc:2258] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:06:45.785205 22070 connection.cc:535] server connection from 127.0.0.1:37196 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:06:45.785688 22386 ts_tablet_manager.cc:1905] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:06:45.786120 21846 connection.cc:535] server connection from 127.0.0.1:45850 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:06:45.788928 22386 ts_tablet_manager.cc:1918] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:45.789043 22386 log.cc:1196] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/08b2e844f6ae42c0b6f8b0818b90cff7
I20260810 23:06:45.789435 22386 ts_tablet_manager.cc:1939] T 08b2e844f6ae42c0b6f8b0818b90cff7 P 521a084e11734cb1b457b9c8f79c08fb: Deleting consensus metadata
W20260810 23:06:45.790809 22211 connection.cc:441] server connection from 127.0.0.1:55448 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55448 (request call id 1) could send its response
I20260810 23:06:45.790997 22383 tablet_replica.cc:330] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: stopping tablet replica
I20260810 23:06:45.791338 22383 raft_consensus.cc:2229] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:45.791723 22383 raft_consensus.cc:2258] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:45.793018 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.773721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.14.75.61:36221)
0810 23:06:45.773840 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.773846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.781854 (+ 8008us) server_negotiation.cc:407] Connection header received
0810 23:06:45.782080 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.782084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.782143 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.782216 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.783434 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.784254 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.785563 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.785894 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.785976 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.789057 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.789062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.789064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.789102 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.790972 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.791516 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.791519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.791522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.791614 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.791908 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.791911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.791913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.792186 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.792331 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.792701 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.792853 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:06:45.793349 22383 ts_tablet_manager.cc:1905] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:45.796128 22383 ts_tablet_manager.cc:1918] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:45.796241 22383 log.cc:1196] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/08b2e844f6ae42c0b6f8b0818b90cff7
I20260810 23:06:45.796567 22383 ts_tablet_manager.cc:1939] T 08b2e844f6ae42c0b6f8b0818b90cff7 P ceb95e4ce6f8428d95cc08d447637af2: Deleting consensus metadata
W20260810 23:06:45.797721 21929 connection.cc:441] server connection from 127.0.0.1:39106 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39106 (request call id 1) could send its response
I20260810 23:06:45.803701 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.773892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.14.75.60:44343)
0810 23:06:45.774007 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.774012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.793038 (+ 19026us) server_negotiation.cc:407] Connection header received
0810 23:06:45.793721 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.793725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.793788 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.793851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.794773 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.795617 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.796870 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.797262 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.797369 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.799356 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.799364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.799368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.799405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.801216 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.801749 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.801753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.801755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.801845 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.802144 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.802147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.802148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.802352 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.802467 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.802849 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.802984 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:06:45.808457 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.798854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.14.75.61:36221)
0810 23:06:45.798997 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.799001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.799087 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:06:45.799268 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.799272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.799318 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.799368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.800202 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.800955 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.802189 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.802527 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.802598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.804647 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.804652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.804654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.804683 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.806573 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.807059 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.807065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.807067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.807163 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.807486 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.807490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.807492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.807748 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.807885 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.808140 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.808295 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:06:45.808488 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:45.798480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.14.75.60:44343)
0810 23:06:45.798607 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:45.798614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:45.798630 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:45.798688 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:45.798691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:45.798742 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:45.798817 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:45.799824 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.800694 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.801967 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:45.802297 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:45.802407 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:45.804462 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:45.804466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:45.804468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:45.804490 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:45.806500 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.807087 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.807090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.807091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.807169 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:45.807575 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:45.807579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:45.807580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:45.807787 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:45.807903 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:45.808254 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 23:06:45.808361 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:06:46.761528 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:46.760951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:59345)
0810 23:06:46.761266 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:46.761351 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":124,"threads_started":1}
I20260810 23:06:46.761983 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:46.761478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:33271)
0810 23:06:46.761786 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:46.761875 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":70,"threads_started":1}
W20260810 23:06:46.762279 22043 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:43701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
W20260810 23:06:46.762351 22326 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:43701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:46.762737 22395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:46.761903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:52733)
0810 23:06:46.762531 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:46.762617 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":64,"threads_started":1}
W20260810 23:06:46.763226 22185 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:43701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.272171 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb837fb6159943aea1dfbdac2f5236d8)
I20260810 23:06:47.272367 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } }
I20260810 23:06:47.273334 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701), 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60:44343)
I20260810 23:06:47.273969 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.273442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34296)
0810 23:06:47.273775 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.273859 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
W20260810 23:06:47.274168 21657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.275009 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.274799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34298)
0810 23:06:47.274906 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.274959 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:06:47.275139 21657 leader_election.cc:334] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701): Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.275404 22401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cb837fb6159943aea1dfbdac2f5236d8)
I20260810 23:06:47.275610 22401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } }
I20260810 23:06:47.276633 22401 leader_election.cc:288] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701), 491c7b699dda43b880c20e2509d2be6a (127.14.75.61:36221)
I20260810 23:06:47.277242 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.276757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34300)
0810 23:06:47.277052 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.277138 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
W20260810 23:06:47.277537 21844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.278249 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.273625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:44343 (local address 127.0.0.1:35222)
0810 23:06:47.273872 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.273884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.274054 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.274379 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.274385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.274413 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.274534 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.274543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.275480 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.275484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.276718 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.276731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.277831 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.277835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.277925 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.277947 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.278030 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.278105 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260810 23:06:47.278287 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.273841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.14.75.60:44343)
0810 23:06:47.274136 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.274143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.274158 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:47.274212 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.274213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.274253 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.274354 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.274674 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.275349 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.276873 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.277582 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.277708 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.277774 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.278070 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.278169 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260810 23:06:47.278644 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.278223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34308)
0810 23:06:47.278521 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.278578 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260810 23:06:47.278785 21844 leader_election.cc:334] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701): Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.279070 21865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "491c7b699dda43b880c20e2509d2be6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" is_pre_election: true
I20260810 23:06:47.279255 21865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491c7b699dda43b880c20e2509d2be6a in term 1.
I20260810 23:06:47.279618 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42b54d7ba53e4fe2a76bf18d3ddd2c93, 491c7b699dda43b880c20e2509d2be6a; no voters: cb837fb6159943aea1dfbdac2f5236d8
I20260810 23:06:47.279853 22396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:06:47.279965 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Starting leader election (detected failure of leader cb837fb6159943aea1dfbdac2f5236d8)
I20260810 23:06:47.280037 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:06:47.281203 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.276778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:36221 (local address 127.0.0.1:42514)
0810 23:06:47.277052 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.277066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.277171 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.277492 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.277495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.277514 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.277625 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.277633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.278766 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.278769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.279953 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.279963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.280839 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.280845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.280947 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.280979 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.281044 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.281097 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":60,"threads_started":1}
I20260810 23:06:47.281324 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.276868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.14.75.61:36221)
0810 23:06:47.277228 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.277237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.277250 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:47.277291 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.277294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.277335 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.277726 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.277821 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.278642 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.280116 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.280703 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.280815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.280868 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.281097 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.281174 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":107,"threads_started":1}
I20260810 23:06:47.281488 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } }
I20260810 23:06:47.281746 21682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "491c7b699dda43b880c20e2509d2be6a" is_pre_election: true
I20260810 23:06:47.281881 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 election: Requested vote from peers cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701), 42b54d7ba53e4fe2a76bf18d3ddd2c93 (127.14.75.60:44343)
I20260810 23:06:47.281972 21682 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42b54d7ba53e4fe2a76bf18d3ddd2c93 in current term 2: Already voted for candidate 491c7b699dda43b880c20e2509d2be6a in this term.
I20260810 23:06:47.282217 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.281994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34316)
0810 23:06:47.282099 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.282153 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:06:47.282181 21865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "491c7b699dda43b880c20e2509d2be6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93"
I20260810 23:06:47.282341 21865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:06:47.282373 21843 leader_election.cc:302] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42b54d7ba53e4fe2a76bf18d3ddd2c93; no voters: 491c7b699dda43b880c20e2509d2be6a, cb837fb6159943aea1dfbdac2f5236d8
I20260810 23:06:47.282931 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.282711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34318)
0810 23:06:47.282824 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.282875 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:06:47.283056 21657 leader_election.cc:334] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701): Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
I20260810 23:06:47.283548 21865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491c7b699dda43b880c20e2509d2be6a in term 2.
I20260810 23:06:47.283677 22401 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:47.283865 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42b54d7ba53e4fe2a76bf18d3ddd2c93, 491c7b699dda43b880c20e2509d2be6a; no voters: cb837fb6159943aea1dfbdac2f5236d8
I20260810 23:06:47.284122 22396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:06:47.284358 22396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [term 2 LEADER]: Becoming Leader. State: Replica: 491c7b699dda43b880c20e2509d2be6a, State: Running, Role: LEADER
I20260810 23:06:47.284626 22396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } }
I20260810 23:06:47.286005 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491c7b699dda43b880c20e2509d2be6a. Latest consensus state: current_term: 2 leader_uuid: "491c7b699dda43b880c20e2509d2be6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:47.286130 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:47.286448 22408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:47.288697 22408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:47.289105 22408 catalog_manager.cc:1259] Loaded cluster ID: 587399e21b03460cab9966f7f29d0af3
I20260810 23:06:47.289152 22408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:47.289933 22408 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:47.290330 22408 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Loaded TSK: 0
I20260810 23:06:47.290997 22408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:47.335311 21671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:47.338474 21671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:47.341754 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.341400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34344)
0810 23:06:47.341552 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.341656 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:06:47.341879 21865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [term 2 FOLLOWER]: Refusing update from remote peer 491c7b699dda43b880c20e2509d2be6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 23:06:47.342471 22406 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 23:06:47.342625 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.342356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.0.0.1:34352)
0810 23:06:47.342490 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.342548 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:06:47.342895 21657 consensus_peers.cc:487] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a -> Peer cb837fb6159943aea1dfbdac2f5236d8 (127.14.75.62:43701): Couldn't send request to peer cb837fb6159943aea1dfbdac2f5236d8. Status: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:06:47.344259 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491c7b699dda43b880c20e2509d2be6a. Latest consensus state: current_term: 2 leader_uuid: "491c7b699dda43b880c20e2509d2be6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:47.344419 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:47.345566 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "491c7b699dda43b880c20e2509d2be6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:47.345752 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:47.346061 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "491c7b699dda43b880c20e2509d2be6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cb837fb6159943aea1dfbdac2f5236d8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43701 } } peers { permanent_uuid: "491c7b699dda43b880c20e2509d2be6a" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 36221 } attrs { promote: false } } peers { permanent_uuid: "42b54d7ba53e4fe2a76bf18d3ddd2c93" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 44343 } attrs { promote: false } } }
I20260810 23:06:47.346189 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b54d7ba53e4fe2a76bf18d3ddd2c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:47.357872 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.353109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.14.75.3:33423)
0810 23:06:47.353373 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.353379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.353392 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:47.353511 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.353514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.353574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.353670 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.353907 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.354749 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.356423 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.357078 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.357265 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.357326 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.357680 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.357743 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260810 23:06:47.357874 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.352815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:33423 (local address 127.0.0.1:52598)
0810 23:06:47.352923 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.352940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.353025 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.353679 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.353682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.353701 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.353790 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.353794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.354862 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.354864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.356264 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.356275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.357214 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.357218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.357549 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.357572 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.357641 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.357719 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":362}
I20260810 23:06:47.358510 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.352461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:45961 (local address 127.0.0.1:47464)
0810 23:06:47.352584 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.352610 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.352693 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.353323 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.353327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.353346 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.353508 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.353516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.354859 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.354863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.356264 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.356275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.357207 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.357211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.357332 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.357359 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.358285 (+ 926us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.358372 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":52}
I20260810 23:06:47.358525 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.352750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.14.75.1:45961)
0810 23:06:47.353031 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.353037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.353052 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:47.353113 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.353116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.353166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.353655 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.353784 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.354689 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.356423 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.357062 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.357190 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.357264 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.358309 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.358405 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260810 23:06:47.358933 22258 tablet_service.cc:1467] Processing CreateTablet for tablet edd386ce7e1945d8aa8f0737299024d9 (DEFAULT_TABLE table=test-table [id=f574da434647476aa4e237a53109c451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:47.359252 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.353232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36827 (local address 127.0.0.1:33074)
0810 23:06:47.354969 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.354981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.355054 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.355478 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.355481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.355498 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.355615 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.355631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.356738 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.356745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.357895 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.357909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.358884 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.358888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.358963 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.358980 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.359044 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.359135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1663,"thread_start_us":103,"threads_started":1}
I20260810 23:06:47.359325 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd386ce7e1945d8aa8f0737299024d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:47.359937 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.353326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.14.75.2:36827)
0810 23:06:47.355246 (+ 1920us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.355250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.355262 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:47.355307 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.355309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.355347 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.355463 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.355800 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.356609 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.358053 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.358755 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.358872 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.358937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.359350 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.359710 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1844,"thread_start_us":88,"threads_started":1}
I20260810 23:06:47.360520 22115 tablet_service.cc:1467] Processing CreateTablet for tablet edd386ce7e1945d8aa8f0737299024d9 (DEFAULT_TABLE table=test-table [id=f574da434647476aa4e237a53109c451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:47.360838 22115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd386ce7e1945d8aa8f0737299024d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:47.362272 22419 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Bootstrap starting.
I20260810 23:06:47.362785 21974 tablet_service.cc:1467] Processing CreateTablet for tablet edd386ce7e1945d8aa8f0737299024d9 (DEFAULT_TABLE table=test-table [id=f574da434647476aa4e237a53109c451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:47.363106 21974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd386ce7e1945d8aa8f0737299024d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:47.363229 22419 tablet_bootstrap.cc:654] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:47.364609 22419 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: No bootstrap required, opened a new log
I20260810 23:06:47.364745 22419 ts_tablet_manager.cc:1397] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:47.365108 22419 raft_consensus.cc:348] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.365227 22419 raft_consensus.cc:374] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:47.365271 22419 raft_consensus.cc:729] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521a084e11734cb1b457b9c8f79c08fb, State: Initialized, Role: FOLLOWER
I20260810 23:06:47.365470 22419 consensus_queue.cc:260] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.365863 22419 ts_tablet_manager.cc:1428] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 23:06:47.366008 22423 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: Bootstrap starting.
I20260810 23:06:47.366349 22420 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Bootstrap starting.
I20260810 23:06:47.367252 22423 tablet_bootstrap.cc:654] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:47.367445 22420 tablet_bootstrap.cc:654] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:47.368916 22328 heartbeater.cc:503] Master 127.14.75.61:36221 was elected leader, sending a full tablet report...
I20260810 23:06:47.368943 22420 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: No bootstrap required, opened a new log
I20260810 23:06:47.369035 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.366285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:59825)
0810 23:06:47.368821 (+ 2536us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.368886 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2467,"thread_start_us":95,"threads_started":1}
I20260810 23:06:47.369038 22420 ts_tablet_manager.cc:1397] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:47.369587 22420 raft_consensus.cc:348] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.369724 22420 raft_consensus.cc:374] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:47.369776 22420 raft_consensus.cc:729] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f3a9e4d53e43a996b53162d58f1c3c, State: Initialized, Role: FOLLOWER
I20260810 23:06:47.369925 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.369456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:53853)
0810 23:06:47.369695 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.369833 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":118,"threads_started":1}
I20260810 23:06:47.369933 22420 consensus_queue.cc:260] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.370339 22420 ts_tablet_manager.cc:1428] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 23:06:47.371021 22423 tablet_bootstrap.cc:492] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: No bootstrap required, opened a new log
I20260810 23:06:47.371152 22423 ts_tablet_manager.cc:1397] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:06:47.371604 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.371053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:33257)
0810 23:06:47.371422 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.371484 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":107,"threads_started":1}
I20260810 23:06:47.371666 22423 raft_consensus.cc:348] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.371795 22423 raft_consensus.cc:374] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:47.371833 22423 raft_consensus.cc:729] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb95e4ce6f8428d95cc08d447637af2, State: Initialized, Role: FOLLOWER
I20260810 23:06:47.371973 22423 consensus_queue.cc:260] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.372054 22186 heartbeater.cc:503] Master 127.14.75.61:36221 was elected leader, sending a full tablet report...
I20260810 23:06:47.372352 22423 ts_tablet_manager.cc:1428] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 23:06:47.372489 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.372051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:41299)
0810 23:06:47.372345 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.372405 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260810 23:06:47.374234 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.373453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:35823)
0810 23:06:47.373997 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.374072 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":103,"threads_started":1}
I20260810 23:06:47.374518 22045 heartbeater.cc:503] Master 127.14.75.61:36221 was elected leader, sending a full tablet report...
I20260810 23:06:47.374914 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.374683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:43021)
0810 23:06:47.374797 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.374851 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:06:47.468151 22422 raft_consensus.cc:482] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:47.468360 22422 raft_consensus.cc:504] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.469673 22422 leader_election.cc:288] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961), 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:47.474004 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.470196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:38541 (local address 127.14.75.1:45961)
0810 23:06:47.470344 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.470350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.470369 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:47.470431 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.470435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.470513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.470573 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.471048 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.471603 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.472937 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.473339 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.473495 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.473569 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.473789 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.473873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:06:47.474045 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.470149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:50373 (local address 127.14.75.2:36827)
0810 23:06:47.470292 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.470298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.470349 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:06:47.470517 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.470520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.470604 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.470691 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:47.471103 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.471691 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.472896 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.473348 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.473541 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.473610 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.473824 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.473906 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:06:47.474037 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.469973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:45961 (local address 127.14.75.3:38541)
0810 23:06:47.470160 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.470181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.470317 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.470621 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.470628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.470653 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.470806 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.470813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.471744 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.471749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.472759 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.472769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.473505 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.473509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.473623 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.473663 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.473736 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.473884 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":46}
I20260810 23:06:47.474412 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.469973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36827 (local address 127.14.75.3:50373)
0810 23:06:47.470253 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.470269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:47.470388 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:47.470736 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:47.470739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:47.470762 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:47.470856 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.470860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.471817 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.471820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.472753 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:47.472762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.473558 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.473562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:47.473656 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.473675 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:06:47.473771 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:06:47.474261 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":568}
I20260810 23:06:47.474649 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd386ce7e1945d8aa8f0737299024d9" candidate_uuid: "521a084e11734cb1b457b9c8f79c08fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb95e4ce6f8428d95cc08d447637af2" is_pre_election: true
I20260810 23:06:47.474884 22136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd386ce7e1945d8aa8f0737299024d9" candidate_uuid: "521a084e11734cb1b457b9c8f79c08fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" is_pre_election: true
I20260810 23:06:47.474939 21995 raft_consensus.cc:2454] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521a084e11734cb1b457b9c8f79c08fb in term 0.
I20260810 23:06:47.475069 22136 raft_consensus.cc:2454] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521a084e11734cb1b457b9c8f79c08fb in term 0.
I20260810 23:06:47.475402 22212 leader_election.cc:302] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521a084e11734cb1b457b9c8f79c08fb, ceb95e4ce6f8428d95cc08d447637af2; no voters:
I20260810 23:06:47.475757 22422 raft_consensus.cc:2781] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:47.475876 22422 raft_consensus.cc:482] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:47.475955 22422 raft_consensus.cc:3037] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:47.477236 22422 raft_consensus.cc:504] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.477727 22422 leader_election.cc:288] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [CANDIDATE]: Term 1 election: Requested vote from peers ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961), 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827)
I20260810 23:06:47.478173 21995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd386ce7e1945d8aa8f0737299024d9" candidate_uuid: "521a084e11734cb1b457b9c8f79c08fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb95e4ce6f8428d95cc08d447637af2"
I20260810 23:06:47.478206 22136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd386ce7e1945d8aa8f0737299024d9" candidate_uuid: "521a084e11734cb1b457b9c8f79c08fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f3a9e4d53e43a996b53162d58f1c3c"
I20260810 23:06:47.478322 21995 raft_consensus.cc:3037] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:47.478343 22136 raft_consensus.cc:3037] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:47.479333 22136 raft_consensus.cc:2454] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521a084e11734cb1b457b9c8f79c08fb in term 1.
I20260810 23:06:47.479382 21995 raft_consensus.cc:2454] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521a084e11734cb1b457b9c8f79c08fb in term 1.
I20260810 23:06:47.479703 22209 leader_election.cc:302] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521a084e11734cb1b457b9c8f79c08fb, 72f3a9e4d53e43a996b53162d58f1c3c; no voters:
I20260810 23:06:47.479921 22422 raft_consensus.cc:2781] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:47.480127 22422 raft_consensus.cc:686] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 1 LEADER]: Becoming Leader. State: Replica: 521a084e11734cb1b457b9c8f79c08fb, State: Running, Role: LEADER
I20260810 23:06:47.480381 22422 consensus_queue.cc:237] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } }
I20260810 23:06:47.480932 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.480620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:56057)
0810 23:06:47.480751 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.480819 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:06:47.482903 21671 catalog_manager.cc:5495] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3). New cstate: current_term: 1 leader_uuid: "521a084e11734cb1b457b9c8f79c08fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521a084e11734cb1b457b9c8f79c08fb" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 33423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:47.513206 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.509540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.14.75.3:33423)
0810 23:06:47.509701 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:47.509706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:47.509728 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:06:47.509800 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:47.509802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:47.509853 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:47.509972 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:47.510221 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.510734 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.511666 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:47.511949 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:47.512059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:47.512337 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:47.512775 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:47.512906 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:47.512987 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:06:47.513043 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:47.516136 22136 raft_consensus.cc:1261] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Refusing update from remote peer 521a084e11734cb1b457b9c8f79c08fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:47.516397 21995 raft_consensus.cc:1261] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Refusing update from remote peer 521a084e11734cb1b457b9c8f79c08fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:47.516722 22422 consensus_queue.cc:1035] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f3a9e4d53e43a996b53162d58f1c3c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:47.516993 22434 consensus_queue.cc:1035] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb95e4ce6f8428d95cc08d447637af2" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:47.519028 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.518724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:57117)
0810 23:06:47.518851 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.518923 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:06:47.519258 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.519002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:35417)
0810 23:06:47.519128 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.519188 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:06:47.522327 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.522032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.1:43049)
0810 23:06:47.522192 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.522247 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 23:06:47.523052 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.522800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.2:58957)
0810 23:06:47.522913 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.522976 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:06:47.524400 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.524119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:43691)
0810 23:06:47.524276 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.524325 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 23:06:47.525111 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.524860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43701 (local address 127.14.75.3:60741)
0810 23:06:47.525010 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.525057 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 23:06:47.532547 21671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:47.532781 21671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:47.537288 21671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 491c7b699dda43b880c20e2509d2be6a: Sending DeleteTablet for 3 replicas of tablet edd386ce7e1945d8aa8f0737299024d9
I20260810 23:06:47.538177 21974 tablet_service.cc:1514] Processing DeleteTablet for tablet edd386ce7e1945d8aa8f0737299024d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:47 UTC) from {username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:47464
I20260810 23:06:47.538199 22258 tablet_service.cc:1514] Processing DeleteTablet for tablet edd386ce7e1945d8aa8f0737299024d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:47 UTC) from {username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:52598
I20260810 23:06:47.538300 22115 tablet_service.cc:1514] Processing DeleteTablet for tablet edd386ce7e1945d8aa8f0737299024d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:47 UTC) from {username='oryx', principal='oryx/127.14.75.61@KRBTEST.COM'} at 127.0.0.1:33074
I20260810 23:06:47.538923 22449 tablet_replica.cc:330] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2: stopping tablet replica
I20260810 23:06:47.539088 22449 raft_consensus.cc:2229] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:47.539235 22449 raft_consensus.cc:2258] T edd386ce7e1945d8aa8f0737299024d9 P ceb95e4ce6f8428d95cc08d447637af2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:47.539602 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21818
I20260810 23:06:47.539887 22450 tablet_replica.cc:330] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: stopping tablet replica
I20260810 23:06:47.540030 22450 raft_consensus.cc:2229] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:47.539584 22451 tablet_replica.cc:330] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: stopping tablet replica
I20260810 23:06:47.540558 22451 raft_consensus.cc:2229] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:47.540741 22451 raft_consensus.cc:2258] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:47.541191 22450 raft_consensus.cc:2258] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:47.542222 22451 ts_tablet_manager.cc:1905] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:47.542235 22450 ts_tablet_manager.cc:1905] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:47.544879 22451 ts_tablet_manager.cc:1918] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:47.545038 22451 log.cc:1196] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/edd386ce7e1945d8aa8f0737299024d9
I20260810 23:06:47.545444 22451 ts_tablet_manager.cc:1939] T edd386ce7e1945d8aa8f0737299024d9 P 72f3a9e4d53e43a996b53162d58f1c3c: Deleting consensus metadata
I20260810 23:06:47.545755 22450 ts_tablet_manager.cc:1918] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:47.545846 22450 log.cc:1196] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/edd386ce7e1945d8aa8f0737299024d9
I20260810 23:06:47.546167 22450 ts_tablet_manager.cc:1939] T edd386ce7e1945d8aa8f0737299024d9 P 521a084e11734cb1b457b9c8f79c08fb: Deleting consensus metadata
I20260810 23:06:47.546473 21656 catalog_manager.cc:4841] TS 72f3a9e4d53e43a996b53162d58f1c3c (127.14.75.2:36827): tablet edd386ce7e1945d8aa8f0737299024d9 (table test-table [id=f574da434647476aa4e237a53109c451]) successfully deleted
I20260810 23:06:47.546931 21657 catalog_manager.cc:4841] TS 521a084e11734cb1b457b9c8f79c08fb (127.14.75.3:33423): tablet edd386ce7e1945d8aa8f0737299024d9 (table test-table [id=f574da434647476aa4e237a53109c451]) successfully deleted
I20260810 23:06:47.550943 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22051
I20260810 23:06:47.551568 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:47.551200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:45961 (local address 127.0.0.1:47474)
0810 23:06:47.551403 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:47.551466 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260810 23:06:47.551921 21659 catalog_manager.cc:4584] TS ceb95e4ce6f8428d95cc08d447637af2 (127.14.75.1:45961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edd386ce7e1945d8aa8f0737299024d9: Network error: Client connection negotiation failed: client connection to 127.14.75.1:45961: connect: Connection refused (error 111)
I20260810 23:06:47.558918 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22191
I20260810 23:06:47.566406 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21533
I20260810 23:06:47.573267 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 21638
2026-08-10T23:06:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5547 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:47 dist-test-slave-srgv krb5kdc[22454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:47 dist-test-slave-srgv krb5kdc[22454](info): set up 2 sockets
Aug 10 23:06:47 dist-test-slave-srgv krb5kdc[22454](info): commencing operation
krb5kdc: starting...
W20260810 23:06:48.639014 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:48 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab.
Entry for principal oryx/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab.
I20260810 23:06:48.766763 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:33817
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab
--principal=oryx/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:48.880295 22467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:48.880590 22467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:48.880635 22467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:48.884797 22467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:48.884868 22467 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:06:48.884886 22467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:48.884904 22467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:48.884920 22467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:48.884935 22467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:48.890487 22467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:33817
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.62.keytab
--principal=oryx/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:48.891665 22467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:48.892727 22467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:48.901672 22472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:48.901705 22473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:48.901893 22467 server_base.cc:1034] running on GCE node
W20260810 23:06:48.901711 22475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:48.902709 22467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:48.903884 22467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:48.905057 22467 hybrid_clock.cc:648] HybridClock initialized: now 1786403208905038 us; error 33 us; skew 500 ppm
Aug 10 23:06:48 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403208, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:48.908052 22467 init.cc:377] Logged in from keytab as oryx/127.14.75.62@KRBTEST.COM (short username oryx)
I20260810 23:06:48.908957 22467 webserver.cc:460] Webserver started at http://127.14.75.62:43155/ using document root <none> and password file <none>
I20260810 23:06:48.909579 22467 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:48.909647 22467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:48.909827 22467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:48.911929 22467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "0992ad29be97438190effcbba5f1f63d"
format_stamp: "Formatted at 2026-08-10 23:06:48 on dist-test-slave-srgv"
server_key: "dfd5453e3e20be50eac9d816be6731ca"
server_key_iv: "a2990f8d984c939872ed36d51b57c6de"
server_key_version: "encryptionkey@0"
I20260810 23:06:48.912462 22467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "0992ad29be97438190effcbba5f1f63d"
format_stamp: "Formatted at 2026-08-10 23:06:48 on dist-test-slave-srgv"
server_key: "dfd5453e3e20be50eac9d816be6731ca"
server_key_iv: "a2990f8d984c939872ed36d51b57c6de"
server_key_version: "encryptionkey@0"
I20260810 23:06:48.916466 22467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:48.919035 22482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:48.920472 22467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:48.920657 22467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "0992ad29be97438190effcbba5f1f63d"
format_stamp: "Formatted at 2026-08-10 23:06:48 on dist-test-slave-srgv"
server_key: "dfd5453e3e20be50eac9d816be6731ca"
server_key_iv: "a2990f8d984c939872ed36d51b57c6de"
server_key_version: "encryptionkey@0"
I20260810 23:06:48.920794 22467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:48.946130 22467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:48.949748 22467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:48.949975 22467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:48.958276 22467 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:33817
I20260810 23:06:48.958304 22547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:33817 every 8 connection(s)
I20260810 23:06:48.959321 22467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:48.962241 22548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:48.964289 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22467
I20260810 23:06:48.964440 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:48.964747 14636 external_mini_cluster.cc:1428] Setting key f5ff6f14140a947ac0e3f23c944d1be0
I20260810 23:06:48.969168 22548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Bootstrap starting.
I20260810 23:06:48.971586 22548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:48.972364 22548 log.cc:824] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:06:48 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:48.974522 22548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: No bootstrap required, opened a new log
I20260810 23:06:48.977636 22548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } }
I20260810 23:06:48.977926 22548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:48.978019 22548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0992ad29be97438190effcbba5f1f63d, State: Initialized, Role: FOLLOWER
I20260810 23:06:48.978550 22548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } }
I20260810 23:06:48.978725 22548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:48.978808 22548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:48.978928 22548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:48.980172 22548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } }
I20260810 23:06:48.980535 22548 leader_election.cc:302] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0992ad29be97438190effcbba5f1f63d; no voters:
I20260810 23:06:48.980719 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:48.966377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.14.75.62:33817)
0810 23:06:48.966861 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:48.966877 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:48.966914 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:48.967600 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:48.967617 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:48.967952 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:48.968300 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:48.969169 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:48.970291 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:48.971189 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:48.971714 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:48.971901 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:48.974497 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:48.974503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:48.974518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:48.974547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:48.977081 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:48.977676 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:48.977681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:48.977686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:48.977805 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:48.978167 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:48.978171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:48.978173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:48.978722 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:48.978998 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:48.979340 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:06:48.979652 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":154,"threads_started":1}
I20260810 23:06:48.980890 22548 leader_election.cc:288] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:48.981144 22553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:48.981367 22553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [term 1 LEADER]: Becoming Leader. State: Replica: 0992ad29be97438190effcbba5f1f63d, State: Running, Role: LEADER
I20260810 23:06:48.981729 22553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } }
I20260810 23:06:48.982232 22548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:48.984194 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0992ad29be97438190effcbba5f1f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } } }
I20260810 23:06:48.984360 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:48.984629 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0992ad29be97438190effcbba5f1f63d. Latest consensus state: current_term: 1 leader_uuid: "0992ad29be97438190effcbba5f1f63d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0992ad29be97438190effcbba5f1f63d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 33817 } } }
I20260810 23:06:48.985167 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d [sys.catalog]: This master's current role is: LEADER
W20260810 23:06:48.987356 22569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:48.987452 22569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:48.987532 22570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:48.988544 22570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:48.993999 22570 catalog_manager.cc:1347] Generated new cluster ID: ea3dd80e1f14403da8c67bdea002cb59
I20260810 23:06:48.994077 22570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:49.014185 22570 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:49.015247 22570 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:49.025875 22570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Generated new TSK 0
I20260810 23:06:49.026659 22570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
I20260810 23:06:49.060213 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33817
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:49.182699 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:49.182987 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:49.183034 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:49.187558 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:49.187628 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:49.187690 22576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:49.187729 22576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:49.193352 22576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:49.194510 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:49.195539 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:49.205646 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:49.205665 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:49.205675 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:49.205894 22576 server_base.cc:1034] running on GCE node
I20260810 23:06:49.206341 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:49.207037 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:49.208246 22576 hybrid_clock.cc:648] HybridClock initialized: now 1786403209208232 us; error 28 us; skew 500 ppm
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:49.211205 22576 init.cc:377] Logged in from keytab as oryx/127.14.75.1@KRBTEST.COM (short username oryx)
I20260810 23:06:49.212033 22576 webserver.cc:460] Webserver started at http://127.14.75.1:44747/ using document root <none> and password file <none>
I20260810 23:06:49.212560 22576 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:49.212618 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:49.212778 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:49.214838 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "cd86e44902ae4493ab9dd886ebf13d80"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "3a7a7d8e5f71538daf3a507afdbcd12b"
server_key_iv: "1f5a27c0542a60badc5582ba224594c4"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.215371 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "cd86e44902ae4493ab9dd886ebf13d80"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "3a7a7d8e5f71538daf3a507afdbcd12b"
server_key_iv: "1f5a27c0542a60badc5582ba224594c4"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.219184 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:06:49.221611 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.222831 22576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:06:49.222930 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "cd86e44902ae4493ab9dd886ebf13d80"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "3a7a7d8e5f71538daf3a507afdbcd12b"
server_key_iv: "1f5a27c0542a60badc5582ba224594c4"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.223019 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:49.242980 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:49.246560 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:49.246726 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:49.248123 22576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:49.248179 22576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.248220 22576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:49.248281 22576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:49.259665 22576 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:42733
I20260810 23:06:49.260532 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:49.261336 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:42733 every 8 connection(s)
I20260810 23:06:49.262806 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.249178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:58495 (local address 127.14.75.62:33817)
0810 23:06:49.249312 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.249316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.250414 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 23:06:49.251635 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.251642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.251715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.251820 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.253449 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.254373 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.255801 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.256048 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.256173 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.259014 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.259018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.259021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.259053 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.260742 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.261392 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.261395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.261397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.261511 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.261841 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.261844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.261845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.262024 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.262104 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.262551 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.262682 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:06:49.263620 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.249768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.1:58495)
0810 23:06:49.250259 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.250304 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.251344 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.252020 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.252030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.252518 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.253237 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.253256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.254528 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.254533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.255671 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.255681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.256233 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.256237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.256438 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.257104 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.257119 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.258837 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.260874 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.260880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.260893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.261269 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.261601 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.261604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.261606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.261751 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.262121 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.262126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.262348 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.262502 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.262765 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":151,"threads_started":1}
I20260810 23:06:49.264876 22711 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33817
I20260810 23:06:49.265044 22711 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:49.265637 22711 heartbeater.cc:511] Master 127.14.75.62:33817 requested a full tablet report, sending...
I20260810 23:06:49.267217 22500 ts_manager.cc:194] Registered new tserver with Master: cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733)
I20260810 23:06:49.268077 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22576
I20260810 23:06:49.268198 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:49.268524 14636 external_mini_cluster.cc:1428] Setting key 105057a4755b79a785107a50d796fb01
I20260810 23:06:49.268832 22500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:58495
I20260810 23:06:49.275733 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.267208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:60801 (local address 127.14.75.62:33817)
0810 23:06:49.267326 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.267329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.267601 (+ 272us) server_negotiation.cc:407] Connection header received
0810 23:06:49.267707 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.267709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.267746 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.267812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.269472 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.270020 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.270935 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.271162 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.271275 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.272494 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.272498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.272501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.272526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.273989 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.274498 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.274501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.274502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.274575 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.274875 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.274877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.274878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.275040 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.275115 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.275476 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.275594 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:06:49.275854 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.267135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.1:60801)
0810 23:06:49.267514 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.267529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.267626 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.268948 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.268952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.269200 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.269325 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.269333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.270141 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.270144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.270809 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.270818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.271280 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.271284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.271380 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.271933 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.271945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.272328 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.274103 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.274110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.274115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.274397 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.274665 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.274671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.274676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.274782 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.275141 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.275147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.275268 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.275396 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.275590 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
I20260810 23:06:49.300107 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33817
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:49.412462 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:49.412737 22720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:49.412812 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:49.416893 22720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:49.416958 22720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:49.416998 22720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:49.417100 22720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:49.422397 22720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:49.423540 22720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:49.424656 22720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:49.434724 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:49.434789 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:49.434724 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:49.434986 22720 server_base.cc:1034] running on GCE node
I20260810 23:06:49.435441 22720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:49.436107 22720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:49.437322 22720 hybrid_clock.cc:648] HybridClock initialized: now 1786403209437286 us; error 52 us; skew 500 ppm
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:49.440213 22720 init.cc:377] Logged in from keytab as oryx/127.14.75.2@KRBTEST.COM (short username oryx)
I20260810 23:06:49.441119 22720 webserver.cc:460] Webserver started at http://127.14.75.2:45053/ using document root <none> and password file <none>
I20260810 23:06:49.441756 22720 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:49.441848 22720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:49.442080 22720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:49.444161 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "8460d2a7b5b2015e674ae6f09b37fd16"
server_key_iv: "a221644ebea4e19f763e97fd98c95846"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.444738 22720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "8460d2a7b5b2015e674ae6f09b37fd16"
server_key_iv: "a221644ebea4e19f763e97fd98c95846"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.448580 22720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 23:06:49.451115 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.452282 22720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:49.452416 22720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "8460d2a7b5b2015e674ae6f09b37fd16"
server_key_iv: "a221644ebea4e19f763e97fd98c95846"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.452534 22720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:49.466681 22720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:49.469614 22720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:49.469841 22720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:49.471315 22720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:49.471398 22720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.471469 22720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:49.471510 22720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:49.482972 22720 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:40023
I20260810 23:06:49.483512 22852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:40023 every 8 connection(s)
I20260810 23:06:49.484182 22720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:49.486783 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.472234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:49695 (local address 127.14.75.62:33817)
0810 23:06:49.472363 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.472367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.473463 (+ 1096us) server_negotiation.cc:407] Connection header received
0810 23:06:49.474520 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.474524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.474614 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.474680 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.476648 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.477579 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.478879 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.479312 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.479448 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.482485 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.482490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.482492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.482526 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.484354 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.485111 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.485114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.485116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.485208 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.485643 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.485647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.485649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.485861 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.485978 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.486480 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.486612 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:06:49.487205 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22720
I20260810 23:06:49.487325 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:49.487638 14636 external_mini_cluster.cc:1428] Setting key ae4af88d9f982b744d60ccdab11dd73c
I20260810 23:06:49.487818 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.472775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.2:49695)
0810 23:06:49.473267 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.473328 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.474234 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.475193 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.475203 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.475759 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.476457 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.476479 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.477743 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.477747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.478772 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.478781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.479499 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.479525 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.479738 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.480503 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.480529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.482287 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.484481 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.484487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.484501 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.484874 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.485321 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.485325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.485327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.485541 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.485976 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.485984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.486234 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.486418 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.486761 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260810 23:06:49.489244 22853 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33817
I20260810 23:06:49.489396 22853 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:49.490031 22853 heartbeater.cc:511] Master 127.14.75.62:33817 requested a full tablet report, sending...
I20260810 23:06:49.491324 22499 ts_manager.cc:194] Registered new tserver with Master: f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023)
I20260810 23:06:49.492094 22499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:49695
WARNING: no policy specified for oryx/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:06:49.500712 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.492042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:43649 (local address 127.14.75.62:33817)
0810 23:06:49.492419 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.492426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.492441 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:49.492477 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.492479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.492518 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.492601 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.493910 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.494530 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.495363 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.495779 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.495907 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.497068 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.497071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.497076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.497108 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.498995 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.499493 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.499500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.499505 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.499605 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.499892 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.499895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.499896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.500109 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.500192 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.500493 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.500592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260810 23:06:49.500762 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.491725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.2:43649)
0810 23:06:49.492043 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.492071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.492214 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.492654 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.492660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.493683 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.493791 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.493797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.494657 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.494661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.495236 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.495244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.495902 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.495905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.495985 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.496542 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.496564 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.496916 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.499105 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.499108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.499109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.499383 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.499710 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.499713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.499714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.499795 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.500194 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.500196 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.500276 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.500358 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.500542 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
I20260810 23:06:49.520252 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:33817
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:49.636747 22862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:49.637102 22862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:49.637193 22862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:49.641744 22862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:49.641860 22862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:49.641942 22862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:06:49.642027 22862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:49.647921 22862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.75.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:49.649171 22862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:49.650274 22862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:49.660419 22870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:49.660553 22862 server_base.cc:1034] running on GCE node
W20260810 23:06:49.660419 22867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:49.660437 22868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:49.661208 22862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:49.662002 22862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:49.663195 22862 hybrid_clock.cc:648] HybridClock initialized: now 1786403209663157 us; error 51 us; skew 500 ppm
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:49.666333 22862 init.cc:377] Logged in from keytab as oryx/127.14.75.3@KRBTEST.COM (short username oryx)
I20260810 23:06:49.667330 22862 webserver.cc:460] Webserver started at http://127.14.75.3:39187/ using document root <none> and password file <none>
I20260810 23:06:49.667934 22862 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:49.668031 22862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:49.668284 22862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:49.670502 22862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "117c0f4043304a58a43ac6d323ae8789"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "ae0f0d88b253c0f90549c4484a36e846"
server_key_iv: "7d518e63866ecf9b71f909ec169c9fdd"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.671084 22862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "117c0f4043304a58a43ac6d323ae8789"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "ae0f0d88b253c0f90549c4484a36e846"
server_key_iv: "7d518e63866ecf9b71f909ec169c9fdd"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.675019 22862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:06:49.677654 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.678896 22862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:49.679054 22862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "117c0f4043304a58a43ac6d323ae8789"
format_stamp: "Formatted at 2026-08-10 23:06:49 on dist-test-slave-srgv"
server_key: "ae0f0d88b253c0f90549c4484a36e846"
server_key_iv: "7d518e63866ecf9b71f909ec169c9fdd"
server_key_version: "encryptionkey@0"
I20260810 23:06:49.679189 22862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:49.707675 22862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:49.710855 22862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:49.711082 22862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:49.712610 22862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:49.712731 22862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:49.712805 22862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:49.712857 22862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:49.724584 22862 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:34067
I20260810 23:06:49.724859 22996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:34067 every 8 connection(s)
I20260810 23:06:49.725983 22862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:49.727725 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 22862
I20260810 23:06:49.727864 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:49.727977 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.713520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:44541 (local address 127.14.75.62:33817)
0810 23:06:49.713652 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.713656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.714771 (+ 1115us) server_negotiation.cc:407] Connection header received
0810 23:06:49.715912 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.715915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.715983 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.716048 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.717904 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.718484 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.719663 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.720008 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.720138 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.723476 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.723482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.723486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.723522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.725387 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.726109 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.726113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.726115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.726211 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.726611 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.726614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.726616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.726891 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.726995 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.727591 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.727768 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:06:49.728206 14636 external_mini_cluster.cc:1428] Setting key 842527a29879ead32f63ee62601cc26c
I20260810 23:06:49.728654 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.714112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.3:44541)
0810 23:06:49.714605 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.714650 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.715635 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.716520 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.716531 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.716980 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.717748 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.717764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.718625 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.718628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.719532 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.719541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.720188 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.720192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.720373 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.721029 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.721045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.723279 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.725575 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.725581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.725596 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.725974 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.726310 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.726312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.726314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.726502 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.727083 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.727095 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.727328 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.727481 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.727782 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
I20260810 23:06:49.730297 22997 heartbeater.cc:347] Connected to a master server at 127.14.75.62:33817
I20260810 23:06:49.730428 22997 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:49.731016 22997 heartbeater.cc:511] Master 127.14.75.62:33817 requested a full tablet report, sending...
I20260810 23:06:49.732122 22499 ts_manager.cc:194] Registered new tserver with Master: 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:49.732875 22499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:44541
I20260810 23:06:49.739403 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.732440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.14.75.3:57751)
0810 23:06:49.732738 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.732750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.732867 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.733094 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.733097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.733314 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:49.733467 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.733476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.734212 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.734215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.735024 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.735033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.735522 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.735524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.735594 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.736159 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:49.736175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:49.736518 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:49.738118 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.738121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.738122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.738329 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.738574 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:49.738576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:49.738577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:49.738650 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:49.738970 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:49.738972 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:49.739049 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.739130 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.739257 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
I20260810 23:06:49.739413 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.732500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:57751 (local address 127.14.75.62:33817)
0810 23:06:49.732633 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.732637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.732821 (+ 184us) server_negotiation.cc:407] Connection header received
0810 23:06:49.732958 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.732961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.733006 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.733079 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.733589 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.734083 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.735192 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.735418 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.735505 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.736641 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.736644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.736647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.736668 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.738013 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.738417 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.738419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.738420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.738496 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.738727 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.738729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.738730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.738892 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.738969 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.739204 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.739299 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:06:49.742239 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:06:49.767805 22551 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:49.760848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.14.75.62:33817)
0810 23:06:49.760998 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.761001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.761014 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:49.761057 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.761061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.761105 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.761204 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.762037 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.762577 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.763463 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.763697 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.763812 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.767623 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45016: BlockingRecv error: recv got EOF from 127.0.0.1:45016 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:06:49 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:49.777266 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.769295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.14.75.62:33817)
0810 23:06:49.769459 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.769464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.769500 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:06:49.769630 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.769633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.769674 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.769749 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:49.770366 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.770873 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.771649 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.771865 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.771945 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.774467 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:49.774470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:49.774472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:49.774495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:49.775791 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.776188 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.776193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.776197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.776306 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:49.776539 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:49.776542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:49.776543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:49.776724 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:49.776833 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.777030 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.777131 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:06:49.780295 22499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:49.783913 22499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:49.799062 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.793031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.14.75.1:42733)
0810 23:06:49.793799 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.793808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.793838 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:06:49.793922 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.793928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.794129 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.794286 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:49.794984 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.796164 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.797772 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798377 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798514 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.798690 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.798818 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.798915 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":106,"threads_started":1}
I20260810 23:06:49.799063 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.792722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42733 (local address 127.0.0.1:39290)
0810 23:06:49.793573 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.793617 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.793757 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.794463 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.794470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.794502 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:49.794707 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.794714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.796284 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.796287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.797619 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.797629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798510 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.798626 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.798663 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.798783 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.798911 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":118,"spinlock_wait_cycles":7040,"thread_start_us":90,"threads_started":1}
I20260810 23:06:49.799649 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.792692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:40023 (local address 127.0.0.1:43336)
0810 23:06:49.793560 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.793616 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.793757 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.794463 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.794470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.794497 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:49.794856 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.794860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.795969 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.795974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.797707 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.797715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798947 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.799318 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.799342 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.799410 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.799485 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":514,"spinlock_wait_cycles":4096,"thread_start_us":100,"threads_started":1}
I20260810 23:06:49.799863 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.793988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.14.75.3:34067)
0810 23:06:49.794280 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.794292 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.794650 (+ 358us) server_negotiation.cc:407] Connection header received
0810 23:06:49.794735 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.794743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.794944 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.795093 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:49.795359 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.796412 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.797796 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798614 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.798880 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.799626 (+ 746us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.799722 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":118,"threads_started":1}
I20260810 23:06:49.800163 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.792825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.14.75.2:40023)
0810 23:06:49.793596 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:49.793609 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:49.793898 (+ 289us) server_negotiation.cc:407] Connection header received
0810 23:06:49.793969 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:49.793973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:49.794164 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:49.794296 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:49.794955 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.795856 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.797805 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798616 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.798880 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:49.799846 (+ 966us) server_negotiation.cc:299] Negotiation successful
0810 23:06:49.800033 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":138,"threads_started":1}
I20260810 23:06:49.799917 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:49.793570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:34067 (local address 127.0.0.1:52560)
0810 23:06:49.794512 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:49.794527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:49.794611 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:49.795083 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:49.795085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:49.795104 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:49.795216 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.795223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.796523 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.796530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.797611 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:49.797634 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:49.798947 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:49.798951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:49.799486 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:49.799505 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:06:49.799577 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:06:49.799645 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":406,"thread_start_us":107,"threads_started":1}
I20260810 23:06:49.802187 22644 tablet_service.cc:1467] Processing CreateTablet for tablet 142519e1e2ed4cab97010865f6ac4533 (DEFAULT_TABLE table=test-table [id=d3f86f19d62f445190e479f7cddd8c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:49.803181 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142519e1e2ed4cab97010865f6ac4533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:49.803347 22787 tablet_service.cc:1467] Processing CreateTablet for tablet 142519e1e2ed4cab97010865f6ac4533 (DEFAULT_TABLE table=test-table [id=d3f86f19d62f445190e479f7cddd8c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:49.803347 22930 tablet_service.cc:1467] Processing CreateTablet for tablet 142519e1e2ed4cab97010865f6ac4533 (DEFAULT_TABLE table=test-table [id=d3f86f19d62f445190e479f7cddd8c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:49.804355 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142519e1e2ed4cab97010865f6ac4533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:49.804355 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142519e1e2ed4cab97010865f6ac4533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:49.809155 23025 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Bootstrap starting.
I20260810 23:06:49.811183 23025 tablet_bootstrap.cc:654] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:49.812048 23025 log.cc:824] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:49.812129 23026 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Bootstrap starting.
I20260810 23:06:49.813521 23027 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Bootstrap starting.
I20260810 23:06:49.814565 23026 tablet_bootstrap.cc:654] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:49.814976 23025 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: No bootstrap required, opened a new log
I20260810 23:06:49.815201 23025 ts_tablet_manager.cc:1397] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 23:06:49.815414 23026 log.cc:824] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:49.815670 23027 tablet_bootstrap.cc:654] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:49.816795 23027 log.cc:824] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:49.817554 23026 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: No bootstrap required, opened a new log
I20260810 23:06:49.817772 23026 ts_tablet_manager.cc:1397] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260810 23:06:49.819123 23027 tablet_bootstrap.cc:492] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: No bootstrap required, opened a new log
I20260810 23:06:49.819329 23027 ts_tablet_manager.cc:1397] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:06:49.819445 23025 raft_consensus.cc:348] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.819723 23025 raft_consensus.cc:374] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:49.819777 23025 raft_consensus.cc:729] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Initialized, Role: FOLLOWER
I20260810 23:06:49.820242 23025 consensus_queue.cc:260] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.820881 22711 heartbeater.cc:503] Master 127.14.75.62:33817 was elected leader, sending a full tablet report...
I20260810 23:06:49.821836 23026 raft_consensus.cc:348] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.822149 23026 raft_consensus.cc:374] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:49.822206 23026 raft_consensus.cc:729] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Initialized, Role: FOLLOWER
I20260810 23:06:49.822692 23026 consensus_queue.cc:260] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.823287 23025 ts_tablet_manager.cc:1428] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 23:06:49.823347 23027 raft_consensus.cc:348] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.823604 23027 raft_consensus.cc:374] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:49.823661 23027 raft_consensus.cc:729] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Initialized, Role: FOLLOWER
I20260810 23:06:49.823786 23026 ts_tablet_manager.cc:1428] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:06:49.824251 23027 consensus_queue.cc:260] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:49.824365 22853 heartbeater.cc:503] Master 127.14.75.62:33817 was elected leader, sending a full tablet report...
I20260810 23:06:49.824987 22997 heartbeater.cc:503] Master 127.14.75.62:33817 was elected leader, sending a full tablet report...
I20260810 23:06:49.825336 23027 ts_tablet_manager.cc:1428] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 23:06:49.977363 22998 tablet.cc:2369] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:49.986033 22854 tablet.cc:2369] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:06:50.012132 22712 tablet.cc:2369] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:50.032306 23031 raft_consensus.cc:482] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:50.032567 23031 raft_consensus.cc:504] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.033879 23031 leader_election.cc:288] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.037955 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.034282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:56805 (local address 127.14.75.2:40023)
0810 23:06:50.034423 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.034430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.034640 (+ 210us) server_negotiation.cc:407] Connection header received
0810 23:06:50.034769 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.034772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.034826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.034907 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.035236 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.035802 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.036911 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.037344 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.037509 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.037590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.037731 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.037813 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:50.037972 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.034140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:40023 (local address 127.14.75.1:56805)
0810 23:06:50.034542 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.034554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.034661 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.034933 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.034936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.034954 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.035101 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.035109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.035915 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.035919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.036781 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.036790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.037489 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.037493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.037605 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.037628 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.037706 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.037788 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":139,"threads_started":1}
I20260810 23:06:50.038111 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.034318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50083 (local address 127.14.75.3:34067)
0810 23:06:50.034455 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.034459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.034799 (+ 340us) server_negotiation.cc:407] Connection header received
0810 23:06:50.034937 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.034940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.034992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.035074 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.035392 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.035915 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.036981 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.037493 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.037599 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.037666 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.037902 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.037982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:06:50.038314 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.034169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:34067 (local address 127.14.75.1:50083)
0810 23:06:50.034666 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.034681 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.034809 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.035094 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.035096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.035110 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.035277 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.035284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.036038 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.036041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.036825 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.036834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.037617 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.037621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.037762 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.037786 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.037864 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.037931 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":117,"threads_started":1}
I20260810 23:06:50.038839 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142519e1e2ed4cab97010865f6ac4533" candidate_uuid: "cd86e44902ae4493ab9dd886ebf13d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" is_pre_election: true
I20260810 23:06:50.038877 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142519e1e2ed4cab97010865f6ac4533" candidate_uuid: "cd86e44902ae4493ab9dd886ebf13d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789" is_pre_election: true
I20260810 23:06:50.039160 22950 raft_consensus.cc:2454] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd86e44902ae4493ab9dd886ebf13d80 in term 0.
I20260810 23:06:50.039163 22807 raft_consensus.cc:2454] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd86e44902ae4493ab9dd886ebf13d80 in term 0.
I20260810 23:06:50.039687 22595 leader_election.cc:302] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.039990 23031 raft_consensus.cc:2781] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:50.040102 23031 raft_consensus.cc:482] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:50.040158 23031 raft_consensus.cc:3037] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.041214 23031 raft_consensus.cc:504] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.041658 23031 leader_election.cc:288] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [CANDIDATE]: Term 1 election: Requested vote from peers f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.042006 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142519e1e2ed4cab97010865f6ac4533" candidate_uuid: "cd86e44902ae4493ab9dd886ebf13d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
I20260810 23:06:50.042099 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142519e1e2ed4cab97010865f6ac4533" candidate_uuid: "cd86e44902ae4493ab9dd886ebf13d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789"
I20260810 23:06:50.042147 22807 raft_consensus.cc:3037] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.042241 22950 raft_consensus.cc:3037] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.043581 22807 raft_consensus.cc:2454] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd86e44902ae4493ab9dd886ebf13d80 in term 1.
I20260810 23:06:50.043928 22595 leader_election.cc:302] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.044047 22950 raft_consensus.cc:2454] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd86e44902ae4493ab9dd886ebf13d80 in term 1.
I20260810 23:06:50.044124 23031 raft_consensus.cc:2781] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:50.044366 23031 raft_consensus.cc:686] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 LEADER]: Becoming Leader. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Running, Role: LEADER
I20260810 23:06:50.044759 23031 consensus_queue.cc:237] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.048405 22499 catalog_manager.cc:5495] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "cd86e44902ae4493ab9dd886ebf13d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:50.106600 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.102782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.14.75.1:42733)
0810 23:06:50.102931 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.102936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.102952 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:50.103095 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.103097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.103158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.103276 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.103553 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.104114 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.105009 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.105235 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.105351 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.105694 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.106166 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.106305 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.106382 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.106446 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:06:50.112896 22807 raft_consensus.cc:1261] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Refusing update from remote peer cd86e44902ae4493ab9dd886ebf13d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.112896 22950 raft_consensus.cc:1261] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Refusing update from remote peer cd86e44902ae4493ab9dd886ebf13d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.113832 23036 consensus_queue.cc:1035] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.114099 23031 consensus_queue.cc:1035] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.122083 23039 mvcc.cc:205] Tried to move back new op lower bound from 7317107548614193152 to 7317107548346331136. Current Snapshot: MvccSnapshot[applied={T|T < 7317107548614193152}]
I20260810 23:06:50.125761 23040 mvcc.cc:205] Tried to move back new op lower bound from 7317107548614193152 to 7317107548346331136. Current Snapshot: MvccSnapshot[applied={T|T < 7317107548614193152}]
I20260810 23:06:50.131346 22499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.131592 22499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.134131 22499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Sending DeleteTablet for 3 replicas of tablet 142519e1e2ed4cab97010865f6ac4533
I20260810 23:06:50.135023 22644 tablet_service.cc:1514] Processing DeleteTablet for tablet 142519e1e2ed4cab97010865f6ac4533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260810 23:06:50.135021 22787 tablet_service.cc:1514] Processing DeleteTablet for tablet 142519e1e2ed4cab97010865f6ac4533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:43336
I20260810 23:06:50.135588 23050 tablet_replica.cc:330] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: stopping tablet replica
I20260810 23:06:50.135622 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet 142519e1e2ed4cab97010865f6ac4533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52560
I20260810 23:06:50.135987 23050 raft_consensus.cc:2229] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.136349 23050 raft_consensus.cc:2258] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.136512 22499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:06:50.136775 23049 tablet_replica.cc:330] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: stopping tablet replica
W20260810 23:06:50.136963 22499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:50.137063 23049 raft_consensus.cc:2229] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:50.137475 23049 raft_consensus.cc:2258] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.138545 23050 ts_tablet_manager.cc:1905] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.138824 23049 ts_tablet_manager.cc:1905] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.141517 23050 ts_tablet_manager.cc:1918] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.141716 23050 log.cc:1196] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/142519e1e2ed4cab97010865f6ac4533
I20260810 23:06:50.141784 23049 ts_tablet_manager.cc:1918] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.141898 23049 log.cc:1196] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/142519e1e2ed4cab97010865f6ac4533
I20260810 23:06:50.142220 23050 ts_tablet_manager.cc:1939] T 142519e1e2ed4cab97010865f6ac4533 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting consensus metadata
I20260810 23:06:50.142254 23049 ts_tablet_manager.cc:1939] T 142519e1e2ed4cab97010865f6ac4533 P cd86e44902ae4493ab9dd886ebf13d80: Deleting consensus metadata
I20260810 23:06:50.143576 22487 catalog_manager.cc:4841] TS f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023): tablet 142519e1e2ed4cab97010865f6ac4533 (table test-table [id=d3f86f19d62f445190e479f7cddd8c91]) successfully deleted
I20260810 23:06:50.143663 22787 tablet_service.cc:1467] Processing CreateTablet for tablet f8686fbeee404e658a91fc91386fbab0 (DEFAULT_TABLE table=test-table [id=e0d453fe7be840adb4974ae5bac550b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.143914 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8686fbeee404e658a91fc91386fbab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.143898 22486 catalog_manager.cc:4841] TS cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733): tablet 142519e1e2ed4cab97010865f6ac4533 (table test-table [id=d3f86f19d62f445190e479f7cddd8c91]) successfully deleted
I20260810 23:06:50.144215 22644 tablet_service.cc:1467] Processing CreateTablet for tablet f8686fbeee404e658a91fc91386fbab0 (DEFAULT_TABLE table=test-table [id=e0d453fe7be840adb4974ae5bac550b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.144460 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8686fbeee404e658a91fc91386fbab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.144692 23051 tablet_replica.cc:330] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: stopping tablet replica
I20260810 23:06:50.145048 23051 raft_consensus.cc:2229] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.145392 23051 raft_consensus.cc:2258] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.146476 22930 tablet_service.cc:1467] Processing CreateTablet for tablet f8686fbeee404e658a91fc91386fbab0 (DEFAULT_TABLE table=test-table [id=e0d453fe7be840adb4974ae5bac550b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.146755 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8686fbeee404e658a91fc91386fbab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.146970 23026 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Bootstrap starting.
I20260810 23:06:50.147975 23026 tablet_bootstrap.cc:654] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.148633 23051 ts_tablet_manager.cc:1905] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.149367 23025 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Bootstrap starting.
I20260810 23:06:50.149670 23027 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Bootstrap starting.
I20260810 23:06:50.150477 23025 tablet_bootstrap.cc:654] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.150586 23027 tablet_bootstrap.cc:654] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.152271 23026 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: No bootstrap required, opened a new log
I20260810 23:06:50.152341 23051 ts_tablet_manager.cc:1918] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.152370 23026 ts_tablet_manager.cc:1397] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260810 23:06:50.152433 23051 log.cc:1196] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/142519e1e2ed4cab97010865f6ac4533
I20260810 23:06:50.152446 23027 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: No bootstrap required, opened a new log
I20260810 23:06:50.152519 23027 ts_tablet_manager.cc:1397] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:50.152951 23027 raft_consensus.cc:348] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.153056 23027 raft_consensus.cc:374] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.153076 23051 ts_tablet_manager.cc:1939] T 142519e1e2ed4cab97010865f6ac4533 P 117c0f4043304a58a43ac6d323ae8789: Deleting consensus metadata
I20260810 23:06:50.153088 23027 raft_consensus.cc:729] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.153455 23027 consensus_queue.cc:260] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.153724 23027 ts_tablet_manager.cc:1428] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.154455 23025 tablet_bootstrap.cc:492] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: No bootstrap required, opened a new log
I20260810 23:06:50.154577 23025 ts_tablet_manager.cc:1397] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:06:50.154520 22484 catalog_manager.cc:4841] TS 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067): tablet 142519e1e2ed4cab97010865f6ac4533 (table test-table [id=d3f86f19d62f445190e479f7cddd8c91]) successfully deleted
I20260810 23:06:50.155084 23025 raft_consensus.cc:348] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.155236 23025 raft_consensus.cc:374] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.155270 23025 raft_consensus.cc:729] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.155418 23025 consensus_queue.cc:260] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.155725 23025 ts_tablet_manager.cc:1428] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:50.152944 23026 raft_consensus.cc:348] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.156028 23026 raft_consensus.cc:374] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.156083 23026 raft_consensus.cc:729] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.156241 23026 consensus_queue.cc:260] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.156520 23026 ts_tablet_manager.cc:1428] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260810 23:06:50.174211 23033 raft_consensus.cc:482] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:50.174378 23033 raft_consensus.cc:504] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.175673 23033 leader_election.cc:288] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023)
I20260810 23:06:50.180471 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.175891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42733 (local address 127.14.75.3:53977)
0810 23:06:50.176007 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.176022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.176116 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.176804 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.176807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.176824 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.176956 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.176963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.177840 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.177845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.179092 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.179102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.179998 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.180002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.180117 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.180147 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.180235 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.180317 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:06:50.180506 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.176380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53977 (local address 127.14.75.1:42733)
0810 23:06:50.176547 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.176552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.176565 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:50.176623 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.176629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.176696 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.176778 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.177112 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.177720 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.179248 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.179869 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.179977 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.180056 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.180275 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.180353 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:06:50.180691 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.176052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:38771 (local address 127.14.75.2:40023)
0810 23:06:50.176188 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.176192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.176480 (+ 288us) server_negotiation.cc:407] Connection header received
0810 23:06:50.176698 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.176701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.176755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.176844 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.177192 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.177989 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.179390 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.179983 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.180048 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.180125 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.180468 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.180557 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:50.181078 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.175891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:40023 (local address 127.14.75.3:38771)
0810 23:06:50.176363 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.176374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.176562 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.176905 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.176909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.176926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.177049 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.177058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.178109 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.178112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.179260 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.179269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.180194 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.180197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.180322 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.180345 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.180846 (+ 501us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.180924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":107,"thread_start_us":98,"threads_started":1}
I20260810 23:06:50.181149 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8686fbeee404e658a91fc91386fbab0" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80" is_pre_election: true
I20260810 23:06:50.181509 22664 raft_consensus.cc:2454] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 0.
I20260810 23:06:50.181598 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8686fbeee404e658a91fc91386fbab0" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" is_pre_election: true
I20260810 23:06:50.181758 22807 raft_consensus.cc:2454] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 0.
I20260810 23:06:50.181947 22881 leader_election.cc:302] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117c0f4043304a58a43ac6d323ae8789, cd86e44902ae4493ab9dd886ebf13d80; no voters:
I20260810 23:06:50.182263 23033 raft_consensus.cc:2781] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:50.182350 23033 raft_consensus.cc:482] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:50.182399 23033 raft_consensus.cc:3037] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.183807 23033 raft_consensus.cc:504] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.184211 23033 leader_election.cc:288] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 election: Requested vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023)
I20260810 23:06:50.184609 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8686fbeee404e658a91fc91386fbab0" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80"
I20260810 23:06:50.184629 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8686fbeee404e658a91fc91386fbab0" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
I20260810 23:06:50.184747 22664 raft_consensus.cc:3037] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.184792 22807 raft_consensus.cc:3037] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.185873 22664 raft_consensus.cc:2454] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 1.
I20260810 23:06:50.186020 22807 raft_consensus.cc:2454] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 1.
I20260810 23:06:50.186244 22881 leader_election.cc:302] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117c0f4043304a58a43ac6d323ae8789, cd86e44902ae4493ab9dd886ebf13d80; no voters:
I20260810 23:06:50.186455 23033 raft_consensus.cc:2781] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:50.186674 23033 raft_consensus.cc:686] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 1 LEADER]: Becoming Leader. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Running, Role: LEADER
I20260810 23:06:50.187139 23033 consensus_queue.cc:237] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.189616 22500 catalog_manager.cc:5495] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "117c0f4043304a58a43ac6d323ae8789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } health_report { overall_health: HEALTHY } } }
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403208, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.62@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:50.233296 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.223416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.14.75.62:33817)
0810 23:06:50.223566 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.223571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.223811 (+ 240us) server_negotiation.cc:407] Connection header received
0810 23:06:50.223939 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.223941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.223982 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.224065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:50.224931 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.225546 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.226650 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.226890 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.226970 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.229877 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:50.229880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:50.229882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:50.229903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:50.231525 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:50.232011 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:50.232017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:50.232019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:50.232095 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:50.232364 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:50.232371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:50.232372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:50.232558 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:50.232668 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.232972 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.233169 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":66}
I20260810 23:06:50.233299 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.223324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:33817 (local address 127.0.0.1:45026)
0810 23:06:50.223682 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.223699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.223846 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.224120 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.224123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.224677 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:50.224798 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.224805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.225651 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.225654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.226521 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.226532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.227010 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.227014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.227103 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.227978 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:50.227993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:50.229717 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:50.231636 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:50.231648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:50.231655 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:50.231905 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:50.232187 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:50.232192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:50.232194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:50.232277 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:50.232675 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:50.232685 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:50.232802 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.232917 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.233093 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260810 23:06:50.237473 22500 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:06:50.238258 22500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:50.243363 22644 tablet_service.cc:1467] Processing CreateTablet for tablet 9d076eb688f440dab0f676514eee0707 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cd0a31edb3a4f7da0a7c5332714f8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:50.243449 22930 tablet_service.cc:1467] Processing CreateTablet for tablet 9d076eb688f440dab0f676514eee0707 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cd0a31edb3a4f7da0a7c5332714f8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:50.243611 22787 tablet_service.cc:1467] Processing CreateTablet for tablet 9d076eb688f440dab0f676514eee0707 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2cd0a31edb3a4f7da0a7c5332714f8d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:06:50.243809 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d076eb688f440dab0f676514eee0707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.243809 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d076eb688f440dab0f676514eee0707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.244000 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d076eb688f440dab0f676514eee0707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.246753 23025 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: Bootstrap starting.
I20260810 23:06:50.247390 23026 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: Bootstrap starting.
I20260810 23:06:50.247813 23025 tablet_bootstrap.cc:654] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.248564 23026 tablet_bootstrap.cc:654] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.248721 23027 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: Bootstrap starting.
I20260810 23:06:50.249773 23027 tablet_bootstrap.cc:654] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.250058 23026 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: No bootstrap required, opened a new log
I20260810 23:06:50.250161 23026 ts_tablet_manager.cc:1397] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 23:06:50.250749 23025 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: No bootstrap required, opened a new log
I20260810 23:06:50.250661 23026 raft_consensus.cc:348] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.250802 23026 raft_consensus.cc:374] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.250842 23025 ts_tablet_manager.cc:1397] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:06:50.250836 23026 raft_consensus.cc:729] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.251324 23025 raft_consensus.cc:348] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.251464 23025 raft_consensus.cc:374] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.251499 23025 raft_consensus.cc:729] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.251526 23026 consensus_queue.cc:260] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.251628 23025 consensus_queue.cc:260] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.251873 23026 ts_tablet_manager.cc:1428] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 23:06:50.251848 23032 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.252012 23032 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.252665 23027 tablet_bootstrap.cc:492] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: No bootstrap required, opened a new log
I20260810 23:06:50.252763 23027 ts_tablet_manager.cc:1397] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:06:50.253242 23027 raft_consensus.cc:348] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.253376 23027 raft_consensus.cc:374] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.253472 23027 raft_consensus.cc:729] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.254038 23027 consensus_queue.cc:260] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.254351 23044 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.254540 23044 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.254746 23033 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.254890 23033 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.255775 23025 ts_tablet_manager.cc:1428] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260810 23:06:50.256069 23027 ts_tablet_manager.cc:1428] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:50.266992 23032 raft_consensus.cc:482] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:50.267156 23032 raft_consensus.cc:504] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.268533 23032 leader_election.cc:288] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.273744 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.268833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:56549 (local address 127.14.75.1:42733)
0810 23:06:50.268971 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.268975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.269352 (+ 377us) server_negotiation.cc:407] Connection header received
0810 23:06:50.269560 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.269563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.269607 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.269652 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.270041 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.270828 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.272460 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.273089 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.273210 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.273278 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.273522 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.273607 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:06:50.274369 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.268767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42733 (local address 127.14.75.2:56549)
0810 23:06:50.269267 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.269281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.269446 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.269792 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.269795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.269810 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.269936 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.269941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.270974 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.270977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.272300 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.272310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.273225 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.273229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.273340 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.273366 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.274127 (+ 761us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.274223 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":136,"threads_started":1}
I20260810 23:06:50.274930 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d076eb688f440dab0f676514eee0707" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80" is_pre_election: true
I20260810 23:06:50.275068 22664 raft_consensus.cc:2454] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 0.
I20260810 23:06:50.275518 22738 leader_election.cc:302] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.275862 23032 raft_consensus.cc:2781] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:50.275918 23032 raft_consensus.cc:482] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:50.275955 23032 raft_consensus.cc:3037] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.277205 23032 raft_consensus.cc:504] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.278096 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d076eb688f440dab0f676514eee0707" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80"
I20260810 23:06:50.278214 22664 raft_consensus.cc:3037] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.279093 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.269330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:57491 (local address 127.14.75.3:34067)
0810 23:06:50.269494 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.269500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.273665 (+ 4165us) server_negotiation.cc:407] Connection header received
0810 23:06:50.273888 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.273892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.273947 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.273999 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:50.274432 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.275213 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.277695 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.278320 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.278407 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.278475 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.278869 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.278963 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:06:50.279373 22664 raft_consensus.cc:2454] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 1.
I20260810 23:06:50.279532 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.268898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:34067 (local address 127.14.75.2:57491)
0810 23:06:50.273558 (+ 4660us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.273576 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.273749 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.274150 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.274153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.274171 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:50.274291 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.274297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.276252 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.276255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.277518 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.277530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.278595 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.278598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.278709 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.278733 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.279298 (+ 565us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.279385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4568,"thread_start_us":129,"threads_started":1}
I20260810 23:06:50.279986 23032 leader_election.cc:288] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 election: Requested vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.280121 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d076eb688f440dab0f676514eee0707" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789" is_pre_election: true
I20260810 23:06:50.280297 22738 leader_election.cc:302] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.280285 22950 raft_consensus.cc:2454] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 0.
I20260810 23:06:50.280272 22949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d076eb688f440dab0f676514eee0707" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789"
I20260810 23:06:50.280449 22949 raft_consensus.cc:3037] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.280656 23032 raft_consensus.cc:2781] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:50.280895 23032 raft_consensus.cc:686] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 LEADER]: Becoming Leader. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Running, Role: LEADER
I20260810 23:06:50.281147 23032 consensus_queue.cc:237] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.281822 22949 raft_consensus.cc:2454] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 1.
I20260810 23:06:50.282318 23032 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: TxnStatusTablet state changed. Reason: New leader f3ce6b9a4401434a87bee9b0e00fa243. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.282476 23032 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:50.282768 23073 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:06:50.282897 23073 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:06:50.283816 22500 catalog_manager.cc:5495] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:50.300699 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.296587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:40023 (local address 127.0.0.1:43352)
0810 23:06:50.296835 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.296851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.297013 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.297345 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.297348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.297359 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:50.297506 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.297513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.298264 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.298270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.298961 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.298970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.299421 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.299425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.299517 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.299570 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.300398 (+ 828us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.300415 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.300488 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.300580 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260810 23:06:50.300699 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.296687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.14.75.2:40023)
0810 23:06:50.296818 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.296823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.296920 (+ 97us) server_negotiation.cc:407] Connection header received
0810 23:06:50.297159 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.297164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.297225 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.297332 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.297624 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.298145 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.299080 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.299310 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.299423 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.299756 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.300297 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.300426 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.300496 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.300552 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:06:50.303812 22949 raft_consensus.cc:1261] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Refusing update from remote peer f3ce6b9a4401434a87bee9b0e00fa243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.304042 22664 raft_consensus.cc:1261] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Refusing update from remote peer f3ce6b9a4401434a87bee9b0e00fa243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.304469 23032 consensus_queue.cc:1035] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.305181 23032 consensus_queue.cc:1035] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.306389 23033 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: TxnStatusTablet state changed. Reason: New leader f3ce6b9a4401434a87bee9b0e00fa243. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.306517 23033 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.306883 23044 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: TxnStatusTablet state changed. Reason: New leader f3ce6b9a4401434a87bee9b0e00fa243. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.307013 23044 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.310283 23070 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.310427 23070 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:50.311473 23033 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.311632 23033 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P 117c0f4043304a58a43ac6d323ae8789: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.312143 23041 mvcc.cc:205] Tried to move back new op lower bound from 7317107549399228416 to 7317107549314523136. Current Snapshot: MvccSnapshot[applied={T|T < 7317107549399228416 or (T in {7317107549399228416})}]
I20260810 23:06:50.315492 23032 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.315636 23032 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P f3ce6b9a4401434a87bee9b0e00fa243: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:06:50.316615 23044 tablet_replica.cc:439] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } } }
I20260810 23:06:50.316764 23044 tablet_replica.cc:442] T 9d076eb688f440dab0f676514eee0707 P cd86e44902ae4493ab9dd886ebf13d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:06:50.320286 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.315746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.14.75.3:34067)
0810 23:06:50.315865 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.315870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.315886 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:50.315969 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.315972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.316033 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.316144 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.316580 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.317372 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.318680 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.319052 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.319176 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.319439 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.319880 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.320009 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.320070 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.320126 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:06:50.332793 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.328877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:33153 (local address 127.14.75.2:40023)
0810 23:06:50.329028 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.329038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.329212 (+ 174us) server_negotiation.cc:407] Connection header received
0810 23:06:50.329350 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.329352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.329397 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.329513 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.329792 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.330392 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.331747 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.331963 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.332046 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.332372 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.332446 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.332499 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.332618 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.332666 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:50.332863 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.328767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:40023 (local address 127.14.75.3:33153)
0810 23:06:50.329121 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.329137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.329262 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.329539 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.329542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.329550 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:50.329683 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.329689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.330489 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.330492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.331580 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.331592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.332084 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.332088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.332187 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.332245 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.332527 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.332534 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.332602 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.332691 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260810 23:06:50.340353 22664 raft_consensus.cc:1261] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Refusing update from remote peer 117c0f4043304a58a43ac6d323ae8789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.340488 22807 raft_consensus.cc:1261] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Refusing update from remote peer 117c0f4043304a58a43ac6d323ae8789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.340906 23033 consensus_queue.cc:1035] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.341323 23056 consensus_queue.cc:1035] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.369297 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.365173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:56887 (local address 127.14.75.3:34067)
0810 23:06:50.365311 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.365316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.365563 (+ 247us) server_negotiation.cc:407] Connection header received
0810 23:06:50.365673 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.365677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.365736 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.365838 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.366126 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.366970 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.368035 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.368391 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.368494 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.368803 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.368889 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.368963 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.369087 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.369147 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:06:50.369347 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.365071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:34067 (local address 127.14.75.2:56887)
0810 23:06:50.365456 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.365472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.365582 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.365851 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:50.365854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:50.365862 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:06:50.365999 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.366005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.367183 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.367186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.367917 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:50.367926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.368528 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.368531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:50.368605 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.368655 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.368986 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.368993 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:06:50.369063 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:06:50.369151 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":152,"threads_started":1}
I20260810 23:06:50.397591 22500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.397789 22500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.400252 22500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Sending DeleteTablet for 3 replicas of tablet f8686fbeee404e658a91fc91386fbab0
I20260810 23:06:50.401180 22644 tablet_service.cc:1514] Processing DeleteTablet for tablet f8686fbeee404e658a91fc91386fbab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260810 23:06:50.401278 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet f8686fbeee404e658a91fc91386fbab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52560
I20260810 23:06:50.401504 23051 tablet_replica.cc:330] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: stopping tablet replica
I20260810 23:06:50.401643 23051 raft_consensus.cc:2229] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:50.401980 23051 raft_consensus.cc:2258] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.402130 23049 tablet_replica.cc:330] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: stopping tablet replica
I20260810 23:06:50.402269 23049 raft_consensus.cc:2229] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.402387 23049 raft_consensus.cc:2258] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.402868 22787 tablet_service.cc:1514] Processing DeleteTablet for tablet f8686fbeee404e658a91fc91386fbab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:43336
I20260810 23:06:50.403080 23050 tablet_replica.cc:330] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: stopping tablet replica
I20260810 23:06:50.403223 23050 raft_consensus.cc:2229] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.403373 23050 raft_consensus.cc:2258] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.403542 23049 ts_tablet_manager.cc:1905] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.404225 23050 ts_tablet_manager.cc:1905] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.406934 23050 ts_tablet_manager.cc:1918] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:50.406939 23051 ts_tablet_manager.cc:1905] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.407012 23050 log.cc:1196] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/f8686fbeee404e658a91fc91386fbab0
I20260810 23:06:50.407289 23050 ts_tablet_manager.cc:1939] T f8686fbeee404e658a91fc91386fbab0 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting consensus metadata
I20260810 23:06:50.408257 22487 catalog_manager.cc:4841] TS f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023): tablet f8686fbeee404e658a91fc91386fbab0 (table test-table [id=e0d453fe7be840adb4974ae5bac550b5]) successfully deleted
I20260810 23:06:50.409610 23051 ts_tablet_manager.cc:1918] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:50.409693 23051 log.cc:1196] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/f8686fbeee404e658a91fc91386fbab0
I20260810 23:06:50.410459 23049 ts_tablet_manager.cc:1918] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:50.410604 23049 log.cc:1196] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/f8686fbeee404e658a91fc91386fbab0
I20260810 23:06:50.410931 23049 ts_tablet_manager.cc:1939] T f8686fbeee404e658a91fc91386fbab0 P cd86e44902ae4493ab9dd886ebf13d80: Deleting consensus metadata
I20260810 23:06:50.411808 22486 catalog_manager.cc:4841] TS cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733): tablet f8686fbeee404e658a91fc91386fbab0 (table test-table [id=e0d453fe7be840adb4974ae5bac550b5]) successfully deleted
I20260810 23:06:50.414003 23051 ts_tablet_manager.cc:1939] T f8686fbeee404e658a91fc91386fbab0 P 117c0f4043304a58a43ac6d323ae8789: Deleting consensus metadata
I20260810 23:06:50.414882 22484 catalog_manager.cc:4841] TS 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067): tablet f8686fbeee404e658a91fc91386fbab0 (table test-table [id=e0d453fe7be840adb4974ae5bac550b5]) successfully deleted
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[22454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:06:50.451156 22551 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:50.442845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.14.75.62:33817)
0810 23:06:50.442988 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.442993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.443213 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:06:50.443360 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.443364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.443420 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.443534 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:50.444481 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.445297 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.446554 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.446907 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.447028 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.450991 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45040: BlockingRecv error: recv got EOF from 127.0.0.1:45040 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[22454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM
I20260810 23:06:50.464898 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.453132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.14.75.62:33817)
0810 23:06:50.453281 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.453285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.453607 (+ 322us) server_negotiation.cc:407] Connection header received
0810 23:06:50.453750 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.453753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.453805 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.453896 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:50.454678 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.455471 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.456671 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.457040 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.457169 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.460991 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:50.460996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:50.460999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:50.461034 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:50.462952 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:50.463487 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:50.463490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:50.463492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:50.463584 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:50.463884 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:50.463887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:50.463889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:50.464134 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:50.464261 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.464600 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.464734 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:50.468330 22500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:50.468845 22500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:50.474524 22644 tablet_service.cc:1467] Processing CreateTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 (DEFAULT_TABLE table=test-table [id=aad156c151bf45d0a5e825aab020a3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.474965 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfd989ca35046bbbd31088d0ba7b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.474926 22787 tablet_service.cc:1467] Processing CreateTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 (DEFAULT_TABLE table=test-table [id=aad156c151bf45d0a5e825aab020a3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.475320 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfd989ca35046bbbd31088d0ba7b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.478428 23025 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Bootstrap starting.
I20260810 23:06:50.478403 22930 tablet_service.cc:1467] Processing CreateTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 (DEFAULT_TABLE table=test-table [id=aad156c151bf45d0a5e825aab020a3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.478736 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfd989ca35046bbbd31088d0ba7b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.479359 23025 tablet_bootstrap.cc:654] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.481091 23027 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Bootstrap starting.
I20260810 23:06:50.482314 23027 tablet_bootstrap.cc:654] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.483233 23026 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Bootstrap starting.
I20260810 23:06:50.484127 23026 tablet_bootstrap.cc:654] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.485023 23027 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: No bootstrap required, opened a new log
I20260810 23:06:50.485126 23027 ts_tablet_manager.cc:1397] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:06:50.485602 23027 raft_consensus.cc:348] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.485731 23027 raft_consensus.cc:374] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.485766 23027 raft_consensus.cc:729] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.485880 23027 consensus_queue.cc:260] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.486444 23026 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: No bootstrap required, opened a new log
I20260810 23:06:50.486536 23026 ts_tablet_manager.cc:1397] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:50.486912 23026 raft_consensus.cc:348] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.487020 23026 raft_consensus.cc:374] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.487049 23026 raft_consensus.cc:729] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.487183 23026 consensus_queue.cc:260] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.487399 23026 ts_tablet_manager.cc:1428] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.488348 23027 ts_tablet_manager.cc:1428] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:50.489753 23025 tablet_bootstrap.cc:492] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: No bootstrap required, opened a new log
I20260810 23:06:50.489871 23025 ts_tablet_manager.cc:1397] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 23:06:50.490311 23025 raft_consensus.cc:348] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.490439 23025 raft_consensus.cc:374] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.490473 23025 raft_consensus.cc:729] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.490594 23025 consensus_queue.cc:260] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.490845 23025 ts_tablet_manager.cc:1428] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.544603 23032 raft_consensus.cc:482] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:50.544800 23032 raft_consensus.cc:504] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.545272 23032 leader_election.cc:288] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.545786 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfd989ca35046bbbd31088d0ba7b087" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80" is_pre_election: true
I20260810 23:06:50.545801 22949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfd989ca35046bbbd31088d0ba7b087" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789" is_pre_election: true
I20260810 23:06:50.545964 22663 raft_consensus.cc:2454] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 0.
I20260810 23:06:50.545989 22949 raft_consensus.cc:2454] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 0.
I20260810 23:06:50.546281 22738 leader_election.cc:302] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.546479 23032 raft_consensus.cc:2781] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:50.546566 23032 raft_consensus.cc:482] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:50.546622 23032 raft_consensus.cc:3037] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.547650 23032 raft_consensus.cc:504] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.548044 23032 leader_election.cc:288] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 election: Requested vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067)
I20260810 23:06:50.548463 22949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfd989ca35046bbbd31088d0ba7b087" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "117c0f4043304a58a43ac6d323ae8789"
I20260810 23:06:50.548619 22949 raft_consensus.cc:3037] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.548617 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfd989ca35046bbbd31088d0ba7b087" candidate_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80"
I20260810 23:06:50.548772 22663 raft_consensus.cc:3037] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.549875 22663 raft_consensus.cc:2454] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 1.
I20260810 23:06:50.549993 22949 raft_consensus.cc:2454] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3ce6b9a4401434a87bee9b0e00fa243 in term 1.
I20260810 23:06:50.550284 22738 leader_election.cc:302] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd86e44902ae4493ab9dd886ebf13d80, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.550590 23032 raft_consensus.cc:2781] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:50.550691 23032 raft_consensus.cc:686] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 LEADER]: Becoming Leader. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Running, Role: LEADER
I20260810 23:06:50.550839 23032 consensus_queue.cc:237] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.552855 22500 catalog_manager.cc:5495] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:50.568460 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.564855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.14.75.2:40023)
0810 23:06:50.564997 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.565002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.565021 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:50.565135 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.565138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.565194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.565285 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.565584 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.566363 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.567226 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.567593 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.567707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.567952 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.568040 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.568122 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.568239 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.568305 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:06:50.570994 22663 raft_consensus.cc:1261] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Refusing update from remote peer f3ce6b9a4401434a87bee9b0e00fa243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.570998 22949 raft_consensus.cc:1261] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Refusing update from remote peer f3ce6b9a4401434a87bee9b0e00fa243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.571513 23070 consensus_queue.cc:1035] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.571686 23032 consensus_queue.cc:1035] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.584601 22500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.584793 22500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.587455 22500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Sending DeleteTablet for 3 replicas of tablet ddfd989ca35046bbbd31088d0ba7b087
I20260810 23:06:50.588104 22644 tablet_service.cc:1514] Processing DeleteTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260810 23:06:50.588220 22787 tablet_service.cc:1514] Processing DeleteTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:43336
I20260810 23:06:50.588450 23049 tablet_replica.cc:330] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: stopping tablet replica
I20260810 23:06:50.588486 23050 tablet_replica.cc:330] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: stopping tablet replica
I20260810 23:06:50.588605 23049 raft_consensus.cc:2229] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.588613 23050 raft_consensus.cc:2229] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:50.588757 23049 raft_consensus.cc:2258] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.589016 23050 raft_consensus.cc:2258] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.589593 23049 ts_tablet_manager.cc:1905] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.589808 23050 ts_tablet_manager.cc:1905] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.591065 22500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:50.591532 22500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:50.592209 23049 ts_tablet_manager.cc:1918] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.592288 23049 log.cc:1196] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/ddfd989ca35046bbbd31088d0ba7b087
I20260810 23:06:50.592414 23050 ts_tablet_manager.cc:1918] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.592494 23050 log.cc:1196] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/ddfd989ca35046bbbd31088d0ba7b087
I20260810 23:06:50.592555 23049 ts_tablet_manager.cc:1939] T ddfd989ca35046bbbd31088d0ba7b087 P cd86e44902ae4493ab9dd886ebf13d80: Deleting consensus metadata
I20260810 23:06:50.592779 23050 ts_tablet_manager.cc:1939] T ddfd989ca35046bbbd31088d0ba7b087 P f3ce6b9a4401434a87bee9b0e00fa243: Deleting consensus metadata
I20260810 23:06:50.593321 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet ddfd989ca35046bbbd31088d0ba7b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52560
I20260810 23:06:50.593552 23051 tablet_replica.cc:330] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: stopping tablet replica
I20260810 23:06:50.593683 23051 raft_consensus.cc:2229] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.593833 23051 raft_consensus.cc:2258] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.594162 22487 catalog_manager.cc:4841] TS f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023): tablet ddfd989ca35046bbbd31088d0ba7b087 (table test-table [id=aad156c151bf45d0a5e825aab020a3f3]) successfully deleted
I20260810 23:06:50.594579 23051 ts_tablet_manager.cc:1905] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.595286 22486 catalog_manager.cc:4841] TS cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733): tablet ddfd989ca35046bbbd31088d0ba7b087 (table test-table [id=aad156c151bf45d0a5e825aab020a3f3]) successfully deleted
I20260810 23:06:50.596894 23051 ts_tablet_manager.cc:1918] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:50.596963 23051 log.cc:1196] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/ddfd989ca35046bbbd31088d0ba7b087
I20260810 23:06:50.597213 23051 ts_tablet_manager.cc:1939] T ddfd989ca35046bbbd31088d0ba7b087 P 117c0f4043304a58a43ac6d323ae8789: Deleting consensus metadata
I20260810 23:06:50.598170 22484 catalog_manager.cc:4841] TS 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067): tablet ddfd989ca35046bbbd31088d0ba7b087 (table test-table [id=aad156c151bf45d0a5e825aab020a3f3]) successfully deleted
I20260810 23:06:50.598713 22930 tablet_service.cc:1467] Processing CreateTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b (DEFAULT_TABLE table=test-table [id=b9f717a9111c470ea103204410c72620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.598968 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455ca9b4356249619e9edd8ab7e8ee0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.598999 22787 tablet_service.cc:1467] Processing CreateTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b (DEFAULT_TABLE table=test-table [id=b9f717a9111c470ea103204410c72620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.599272 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455ca9b4356249619e9edd8ab7e8ee0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.601218 23027 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Bootstrap starting.
I20260810 23:06:50.602077 23027 tablet_bootstrap.cc:654] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.602041 22644 tablet_service.cc:1467] Processing CreateTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b (DEFAULT_TABLE table=test-table [id=b9f717a9111c470ea103204410c72620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:50.602313 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 455ca9b4356249619e9edd8ab7e8ee0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:50.604679 23025 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: Bootstrap starting.
I20260810 23:06:50.605464 23026 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Bootstrap starting.
I20260810 23:06:50.605521 23025 tablet_bootstrap.cc:654] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.606266 23026 tablet_bootstrap.cc:654] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:50.607831 23026 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: No bootstrap required, opened a new log
I20260810 23:06:50.607930 23026 ts_tablet_manager.cc:1397] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:06:50.608372 23026 raft_consensus.cc:348] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.608503 23026 raft_consensus.cc:374] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.608536 23026 raft_consensus.cc:729] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ce6b9a4401434a87bee9b0e00fa243, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.608671 23026 consensus_queue.cc:260] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.608939 23026 ts_tablet_manager.cc:1428] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:50.609872 23027 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: No bootstrap required, opened a new log
I20260810 23:06:50.609979 23027 ts_tablet_manager.cc:1397] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260810 23:06:50.610391 23027 raft_consensus.cc:348] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.610510 23027 raft_consensus.cc:374] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.610545 23027 raft_consensus.cc:729] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.610661 23027 consensus_queue.cc:260] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.610906 23027 ts_tablet_manager.cc:1428] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.611778 23025 tablet_bootstrap.cc:492] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: No bootstrap required, opened a new log
I20260810 23:06:50.611891 23025 ts_tablet_manager.cc:1397] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260810 23:06:50.612286 23025 raft_consensus.cc:348] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.612385 23025 raft_consensus.cc:374] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:50.612406 23025 raft_consensus.cc:729] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd86e44902ae4493ab9dd886ebf13d80, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.612586 23025 consensus_queue.cc:260] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.612846 23025 ts_tablet_manager.cc:1428] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:50.721236 23033 raft_consensus.cc:482] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:50.721450 23033 raft_consensus.cc:504] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.721951 23033 leader_election.cc:288] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023)
I20260810 23:06:50.722510 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455ca9b4356249619e9edd8ab7e8ee0b" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80" is_pre_election: true
I20260810 23:06:50.722510 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455ca9b4356249619e9edd8ab7e8ee0b" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" is_pre_election: true
I20260810 23:06:50.722674 22663 raft_consensus.cc:2454] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 0.
I20260810 23:06:50.722674 22807 raft_consensus.cc:2454] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 0.
I20260810 23:06:50.723024 22882 leader_election.cc:302] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117c0f4043304a58a43ac6d323ae8789, f3ce6b9a4401434a87bee9b0e00fa243; no voters:
I20260810 23:06:50.723212 23033 raft_consensus.cc:2781] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:50.723282 23033 raft_consensus.cc:482] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:50.723340 23033 raft_consensus.cc:3037] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.724478 23033 raft_consensus.cc:504] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.724879 23033 leader_election.cc:288] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 election: Requested vote from peers cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733), f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023)
I20260810 23:06:50.725466 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455ca9b4356249619e9edd8ab7e8ee0b" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd86e44902ae4493ab9dd886ebf13d80"
I20260810 23:06:50.725607 22663 raft_consensus.cc:3037] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.726814 22663 raft_consensus.cc:2454] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 1.
I20260810 23:06:50.727139 22881 leader_election.cc:302] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117c0f4043304a58a43ac6d323ae8789, cd86e44902ae4493ab9dd886ebf13d80; no voters:
I20260810 23:06:50.727222 22807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "455ca9b4356249619e9edd8ab7e8ee0b" candidate_uuid: "117c0f4043304a58a43ac6d323ae8789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ce6b9a4401434a87bee9b0e00fa243"
I20260810 23:06:50.727366 22807 raft_consensus.cc:3037] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:50.727403 23033 raft_consensus.cc:2781] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:50.727478 23033 raft_consensus.cc:686] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 1 LEADER]: Becoming Leader. State: Replica: 117c0f4043304a58a43ac6d323ae8789, State: Running, Role: LEADER
I20260810 23:06:50.727627 23033 consensus_queue.cc:237] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } }
I20260810 23:06:50.728452 22807 raft_consensus.cc:2454] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117c0f4043304a58a43ac6d323ae8789 in term 1.
I20260810 23:06:50.730254 22501 catalog_manager.cc:5495] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 reported cstate change: term changed from 0 to 1, leader changed from <none> to 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "117c0f4043304a58a43ac6d323ae8789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117c0f4043304a58a43ac6d323ae8789" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 34067 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:50.747629 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.744320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.14.75.3:34067)
0810 23:06:50.744437 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:50.744444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:50.744460 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:50.744638 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:50.744641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:50.744691 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:50.744780 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:50.745041 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.745691 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.746541 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:50.746777 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:50.746885 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:50.747160 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:50.747248 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:50.747331 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:50.747426 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:06:50.747481 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:06:50.754475 22807 raft_consensus.cc:1261] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Refusing update from remote peer 117c0f4043304a58a43ac6d323ae8789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.754942 23033 consensus_queue.cc:1035] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3ce6b9a4401434a87bee9b0e00fa243" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.755075 22663 raft_consensus.cc:1261] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Refusing update from remote peer 117c0f4043304a58a43ac6d323ae8789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:50.755427 23033 consensus_queue.cc:1035] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd86e44902ae4493ab9dd886ebf13d80" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:50.789029 22500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.789186 22500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:50.791463 22500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0992ad29be97438190effcbba5f1f63d: Sending DeleteTablet for 3 replicas of tablet 455ca9b4356249619e9edd8ab7e8ee0b
I20260810 23:06:50.792086 22787 tablet_service.cc:1514] Processing DeleteTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:43336
I20260810 23:06:50.792222 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:52560
I20260810 23:06:50.792238 22644 tablet_service.cc:1514] Processing DeleteTablet for tablet 455ca9b4356249619e9edd8ab7e8ee0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:50 UTC) from {username='oryx', principal='oryx/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260810 23:06:50.792306 23050 tablet_replica.cc:330] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: stopping tablet replica
I20260810 23:06:50.792423 23050 raft_consensus.cc:2229] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.792440 23049 tablet_replica.cc:330] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80: stopping tablet replica
I20260810 23:06:50.792557 23050 raft_consensus.cc:2258] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.792582 23049 raft_consensus.cc:2229] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:50.792713 23049 raft_consensus.cc:2258] T 455ca9b4356249619e9edd8ab7e8ee0b P cd86e44902ae4493ab9dd886ebf13d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.792765 23051 tablet_replica.cc:330] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: stopping tablet replica
I20260810 23:06:50.792909 23051 raft_consensus.cc:2229] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:50.792919 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22576
I20260810 23:06:50.793146 23051 raft_consensus.cc:2258] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:50.795186 23051 ts_tablet_manager.cc:1905] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.795655 23050 ts_tablet_manager.cc:1905] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:50.797967 23051 ts_tablet_manager.cc:1918] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:50.798076 23051 log.cc:1196] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/455ca9b4356249619e9edd8ab7e8ee0b
I20260810 23:06:50.798261 23050 ts_tablet_manager.cc:1918] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:06:50.798349 23050 log.cc:1196] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/455ca9b4356249619e9edd8ab7e8ee0b
I20260810 23:06:50.798380 23051 ts_tablet_manager.cc:1939] T 455ca9b4356249619e9edd8ab7e8ee0b P 117c0f4043304a58a43ac6d323ae8789: Deleting consensus metadata
I20260810 23:06:50.798614 23050 ts_tablet_manager.cc:1939] T 455ca9b4356249619e9edd8ab7e8ee0b P f3ce6b9a4401434a87bee9b0e00fa243: Deleting consensus metadata
I20260810 23:06:50.799274 22484 catalog_manager.cc:4841] TS 117c0f4043304a58a43ac6d323ae8789 (127.14.75.3:34067): tablet 455ca9b4356249619e9edd8ab7e8ee0b (table test-table [id=b9f717a9111c470ea103204410c72620]) successfully deleted
I20260810 23:06:50.799356 22487 catalog_manager.cc:4841] TS f3ce6b9a4401434a87bee9b0e00fa243 (127.14.75.2:40023): tablet 455ca9b4356249619e9edd8ab7e8ee0b (table test-table [id=b9f717a9111c470ea103204410c72620]) successfully deleted
W20260810 23:06:50.802055 22486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:42733 (error 108)
I20260810 23:06:50.802877 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22720
I20260810 23:06:50.803224 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.802761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42733 (local address 127.0.0.1:39308)
0810 23:06:50.803007 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.803095 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:42733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":50,"threads_started":1}
W20260810 23:06:50.804564 22486 catalog_manager.cc:4584] TS cd86e44902ae4493ab9dd886ebf13d80 (127.14.75.1:42733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 455ca9b4356249619e9edd8ab7e8ee0b: Network error: Client connection negotiation failed: client connection to 127.14.75.1:42733: connect: Connection refused (error 111)
I20260810 23:06:50.811808 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22862
I20260810 23:06:50.819689 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 22467
2026-08-10T23:06:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3245 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[23119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[23119](info): set up 2 sockets
Aug 10 23:06:50 dist-test-slave-srgv krb5kdc[23119](info): commencing operation
krb5kdc: starting...
W20260810 23:06:51.894856 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:51 dist-test-slave-srgv krb5kdc[23119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:52.028424 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40389
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:52.143242 23132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:52.143659 23132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:52.143745 23132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:52.148161 23132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:52.148283 23132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:52.148322 23132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:52.148362 23132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:52.148418 23132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:52.154248 23132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:52.155442 23132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:52.156520 23132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:52.165500 23137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:52.165522 23138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:52.165642 23132 server_base.cc:1034] running on GCE node
W20260810 23:06:52.165504 23140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:52.166615 23132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:52.167933 23132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:52.169148 23132 hybrid_clock.cc:648] HybridClock initialized: now 1786403212169119 us; error 47 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:06:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1355 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:52 dist-test-slave-srgv krb5kdc[23145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:52 dist-test-slave-srgv krb5kdc[23145](info): set up 2 sockets
Aug 10 23:06:52 dist-test-slave-srgv krb5kdc[23145](info): commencing operation
krb5kdc: starting...
W20260810 23:06:53.247363 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:06:53 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:06:53.380967 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41391
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:53.499050 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:53.499395 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:53.499465 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:53.503737 23158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:53.503844 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:53.503880 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:53.503923 23158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:53.503960 23158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:53.509793 23158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:53.510967 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:53.512048 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:53.521111 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:53.521111 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:53.521309 23158 server_base.cc:1034] running on GCE node
W20260810 23:06:53.521103 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:53.522133 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:53.523358 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:53.524549 23158 hybrid_clock.cc:648] HybridClock initialized: now 1786403213524523 us; error 45 us; skew 500 ppm
Aug 10 23:06:53 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:53.531778 23158 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:53.533063 23158 webserver.cc:460] Webserver started at http://127.14.75.62:39575/ using document root <none> and password file <none>
I20260810 23:06:53.533792 23158 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:53.533895 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:53.534148 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:53.536326 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "377450e3f22a45d488a175846f3f3bea"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "f3313d669aebcd3a29515d5b6451d249"
server_key_iv: "aec6d5619a69442377866e075543101c"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.536924 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "377450e3f22a45d488a175846f3f3bea"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "f3313d669aebcd3a29515d5b6451d249"
server_key_iv: "aec6d5619a69442377866e075543101c"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.540959 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:53.543695 23173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:53.545099 23158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:53.545279 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "377450e3f22a45d488a175846f3f3bea"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "f3313d669aebcd3a29515d5b6451d249"
server_key_iv: "aec6d5619a69442377866e075543101c"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.545411 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:53.583262 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:53.586429 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:53.586658 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:53.594187 23158 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41391
I20260810 23:06:53.594209 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41391 every 8 connection(s)
I20260810 23:06:53.595297 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:53.598304 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:53.598941 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23158
I20260810 23:06:53.599090 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:06:53.599421 14636 external_mini_cluster.cc:1428] Setting key d91b174cb0c1e710037b77714e7bf863
I20260810 23:06:53.604707 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Bootstrap starting.
I20260810 23:06:53.607573 23227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:53.608588 23227 log.cc:824] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:06:53 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:53.610801 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: No bootstrap required, opened a new log
I20260810 23:06:53.614758 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.600964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.14.75.62:41391)
0810 23:06:53.601475 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:53.601491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:53.601527 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:06:53.602178 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:53.602194 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:53.602520 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:53.602877 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:53.603719 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.604722 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.606022 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.606487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.606608 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:53.609814 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:53.609819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:53.609833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:53.609859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:53.611707 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:53.612236 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:53.612241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:53.612246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:53.612369 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:53.612694 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:53.612696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:53.612697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:53.613111 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:53.613266 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:53.613601 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:06:53.613945 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260810 23:06:53.615172 23227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:53.615437 23227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:53.615504 23227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377450e3f22a45d488a175846f3f3bea, State: Initialized, Role: FOLLOWER
I20260810 23:06:53.615942 23227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:53.616101 23227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:53.616195 23227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:53.616324 23227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:53.617338 23227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:53.617712 23227 leader_election.cc:302] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 377450e3f22a45d488a175846f3f3bea; no voters:
I20260810 23:06:53.618060 23227 leader_election.cc:288] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:53.618201 23232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:53.618459 23232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 LEADER]: Becoming Leader. State: Replica: 377450e3f22a45d488a175846f3f3bea, State: Running, Role: LEADER
I20260810 23:06:53.618788 23232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:53.619351 23227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:53.620708 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "377450e3f22a45d488a175846f3f3bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } } }
I20260810 23:06:53.620749 23234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 377450e3f22a45d488a175846f3f3bea. Latest consensus state: current_term: 1 leader_uuid: "377450e3f22a45d488a175846f3f3bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } } }
I20260810 23:06:53.620836 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:53.620874 23234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:53.621268 23242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:53.624545 23242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:53.631031 23242 catalog_manager.cc:1347] Generated new cluster ID: 19b4b57fa0284dfe97413577d379c1b9
I20260810 23:06:53.631120 23242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:53.650005 23242 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:53.651021 23242 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:53.655964 23242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Generated new TSK 0
I20260810 23:06:53.656895 23242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:06:53.696784 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:53.814276 23254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:53.814577 23254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:53.814656 23254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:53.818943 23254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:53.819041 23254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:53.819162 23254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:53.825214 23254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:53.826462 23254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:53.827557 23254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:53.837692 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:53.837708 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:53.837718 23262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:53.838161 23254 server_base.cc:1034] running on GCE node
I20260810 23:06:53.838620 23254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:53.839329 23254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:53.840523 23254 hybrid_clock.cc:648] HybridClock initialized: now 1786403213840511 us; error 34 us; skew 500 ppm
Aug 10 23:06:53 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:53.843505 23254 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:06:53.844632 23254 webserver.cc:460] Webserver started at http://127.14.75.1:33259/ using document root <none> and password file <none>
I20260810 23:06:53.845522 23254 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:53.845602 23254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:53.845793 23254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:53.847903 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "061d580a864d433db8089b41e851ee9b"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "ed6220cf20a3c267f0e495d0e96b9d1b"
server_key_iv: "e33b25377071cac3d329ff68247855f2"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.848464 23254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "061d580a864d433db8089b41e851ee9b"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "ed6220cf20a3c267f0e495d0e96b9d1b"
server_key_iv: "e33b25377071cac3d329ff68247855f2"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.852607 23254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 23:06:53.855283 23270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:53.856658 23254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:53.856812 23254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "061d580a864d433db8089b41e851ee9b"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "ed6220cf20a3c267f0e495d0e96b9d1b"
server_key_iv: "e33b25377071cac3d329ff68247855f2"
server_key_version: "encryptionkey@0"
I20260810 23:06:53.856920 23254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:53.874028 23254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:53.877799 23254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:53.877983 23254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:53.878619 23254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:53.879449 23254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:53.879498 23254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:53.879537 23254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:53.879561 23254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:53.889138 23254 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:44083
I20260810 23:06:53.889227 23386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:44083 every 8 connection(s)
I20260810 23:06:53.890170 23254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:53.894392 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23254
I20260810 23:06:53.894552 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:53.894905 14636 external_mini_cluster.cc:1428] Setting key c7480ae50a89e84ddacebffac341b731
Aug 10 23:06:53 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:06:53.905690 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.892092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50705 (local address 127.14.75.62:41391)
0810 23:06:53.892287 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:53.892292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:53.893150 (+ 858us) server_negotiation.cc:407] Connection header received
0810 23:06:53.894535 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:53.894539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:53.894607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:53.894711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:53.896106 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.896777 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.898051 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.898313 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.898431 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:53.901587 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:53.901591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:53.901593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:53.901618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:53.903158 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:53.903858 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:53.903861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:53.903862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:53.903937 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:53.904386 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:53.904388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:53.904389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:53.904570 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:53.904643 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:53.905373 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 23:06:53.905550 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:06:53.906692 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.892501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:50705)
0810 23:06:53.892974 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:53.893020 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:53.894291 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:53.894884 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:53.894891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:53.895386 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:53.895952 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:53.895967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.896923 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.896926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:53.897919 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:53.897929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:53.898479 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:53.898483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:53.898667 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:53.899356 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:53.899373 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:53.901356 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:53.903303 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:53.903314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:53.903334 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:53.903742 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:53.904042 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:53.904046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:53.904050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:53.904279 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:53.904687 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:53.904698 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:53.905058 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 23:06:53.905316 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 23:06:53.905707 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:06:53.908124 23387 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41391
I20260810 23:06:53.908402 23387 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:53.908936 23387 heartbeater.cc:511] Master 127.14.75.62:41391 requested a full tablet report, sending...
I20260810 23:06:53.910557 23191 ts_manager.cc:194] Registered new tserver with Master: 061d580a864d433db8089b41e851ee9b (127.14.75.1:44083)
I20260810 23:06:53.911736 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:50705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:06:53.929553 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:54.050947 23392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:54.051285 23392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:54.051360 23392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:54.055606 23392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:54.055725 23392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:54.055823 23392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:54.061690 23392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:54.062845 23392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:54.063894 23392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:54.074249 23398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.074280 23397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.074259 23400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:54.074659 23392 server_base.cc:1034] running on GCE node
I20260810 23:06:54.075160 23392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:54.075855 23392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:54.077047 23392 hybrid_clock.cc:648] HybridClock initialized: now 1786403214077027 us; error 39 us; skew 500 ppm
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:54.080085 23392 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:06:54.081024 23392 webserver.cc:460] Webserver started at http://127.14.75.2:37223/ using document root <none> and password file <none>
I20260810 23:06:54.081700 23392 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:54.081802 23392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:54.082041 23392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:54.084800 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "5a827b23b5ea432186cd4bca52929b90"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "050dc992d2b3a1ee84008e6287392423"
server_key_iv: "b10577f3b0bb01290ff2753abd751234"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.085631 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "5a827b23b5ea432186cd4bca52929b90"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "050dc992d2b3a1ee84008e6287392423"
server_key_iv: "b10577f3b0bb01290ff2753abd751234"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.091279 23392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:06:54.094385 23407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.095719 23392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:54.095850 23392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "5a827b23b5ea432186cd4bca52929b90"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "050dc992d2b3a1ee84008e6287392423"
server_key_iv: "b10577f3b0bb01290ff2753abd751234"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.095954 23392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:54.111709 23392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:54.114912 23392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:54.115147 23392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:54.115782 23392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:54.116643 23392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:54.116701 23392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.116771 23392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:54.116808 23392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.126295 23392 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36479
I20260810 23:06:54.126356 23522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36479 every 8 connection(s)
I20260810 23:06:54.127295 23392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:54.137383 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23392
I20260810 23:06:54.137557 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:54.137916 14636 external_mini_cluster.cc:1428] Setting key 2f27e3b8f8998bc4ae2aa448ad130e09
I20260810 23:06:54.142622 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.129190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:40795 (local address 127.14.75.62:41391)
0810 23:06:54.129335 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:54.129339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:54.130266 (+ 927us) server_negotiation.cc:407] Connection header received
0810 23:06:54.131842 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:54.131846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:54.131913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:54.132000 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:54.133252 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.133895 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.135098 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.135346 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.135462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.138410 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:54.138413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:54.138415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:54.138442 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:54.140367 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:54.140928 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:54.140935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:54.140940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:54.141039 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:54.141451 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:54.141456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:54.141457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:54.141661 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:54.141772 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:54.142322 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 23:06:54.142460 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:06:54.143520 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.129597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:40795)
0810 23:06:54.130100 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.130144 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:54.131616 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:54.132204 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:54.132211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:54.132591 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:54.133090 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.133103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.134028 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.134032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.134963 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.134972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.135512 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.135517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.135696 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.136422 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:54.136438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:54.138199 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:54.140501 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:54.140506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:54.140522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:54.140820 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:54.141152 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:54.141154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:54.141155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:54.141314 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:54.141787 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:54.141796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:54.142071 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 23:06:54.142281 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 23:06:54.142640 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260810 23:06:54.144833 23523 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41391
I20260810 23:06:54.145123 23523 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:54.145716 23523 heartbeater.cc:511] Master 127.14.75.62:41391 requested a full tablet report, sending...
I20260810 23:06:54.146834 23191 ts_manager.cc:194] Registered new tserver with Master: 5a827b23b5ea432186cd4bca52929b90 (127.14.75.2:36479)
I20260810 23:06:54.147433 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:40795
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:06:54.166864 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:54.284248 23528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:54.284554 23528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:54.284602 23528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:54.288705 23528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:54.288793 23528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:54.288856 23528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:54.294458 23528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:54.295564 23528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:54.296558 23528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:54.306784 23533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:54.306901 23528 server_base.cc:1034] running on GCE node
W20260810 23:06:54.306787 23534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.306824 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:54.307541 23528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:54.308168 23528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:54.309336 23528 hybrid_clock.cc:648] HybridClock initialized: now 1786403214309302 us; error 47 us; skew 500 ppm
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:54.312311 23528 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:06:54.313230 23528 webserver.cc:460] Webserver started at http://127.14.75.3:36541/ using document root <none> and password file <none>
I20260810 23:06:54.313866 23528 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:54.313935 23528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:54.314111 23528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:54.316237 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "f185d287acfe44b9b459a4f94d429b57"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "9328306fe8ea67665bb0ed6ed78ded9b"
server_key_iv: "7f60f57df1d74e83e996a3e2cd06bbd4"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.316807 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "f185d287acfe44b9b459a4f94d429b57"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "9328306fe8ea67665bb0ed6ed78ded9b"
server_key_iv: "7f60f57df1d74e83e996a3e2cd06bbd4"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.321242 23528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 23:06:54.323956 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.325291 23528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:54.325477 23528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "f185d287acfe44b9b459a4f94d429b57"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "9328306fe8ea67665bb0ed6ed78ded9b"
server_key_iv: "7f60f57df1d74e83e996a3e2cd06bbd4"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.325618 23528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:54.339887 23528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:54.343288 23528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:54.343533 23528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:54.344216 23528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:54.345127 23528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:54.345206 23528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.345288 23528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:54.345340 23528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.355113 23528 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:34063
I20260810 23:06:54.355158 23661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:34063 every 8 connection(s)
I20260810 23:06:54.356086 23528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:54.364506 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23528
I20260810 23:06:54.364687 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:54.364964 14636 external_mini_cluster.cc:1428] Setting key b9021a45c2c04d4c719ac744fda7c7b1
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:06:54.370942 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.357980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:51443 (local address 127.14.75.62:41391)
0810 23:06:54.358107 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:54.358111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:54.359017 (+ 906us) server_negotiation.cc:407] Connection header received
0810 23:06:54.360462 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:54.360465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:54.360533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:54.360632 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:54.362393 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.362936 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.364178 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.364398 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.364516 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.367640 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:54.367643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:54.367645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:54.367672 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:54.368979 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:54.369524 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:54.369528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:54.369529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:54.369630 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:54.369953 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:54.369955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:54.369957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:54.370143 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:54.370219 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:54.370709 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 23:06:54.370820 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:06:54.372177 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.358386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:51443)
0810 23:06:54.358852 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.358898 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:54.360185 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:54.360900 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:54.360910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:54.361657 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:54.362233 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.362247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.363101 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.363104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.363994 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.364003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.364565 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.364569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.364812 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.365549 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:54.365574 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:54.367486 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:54.369094 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:54.369099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:54.369112 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:54.369389 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:54.369720 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:54.369722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:54.369723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:54.369865 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:54.370235 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:54.370240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:54.370490 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 23:06:54.370645 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 23:06:54.370926 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260810 23:06:54.373482 23662 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41391
I20260810 23:06:54.373747 23662 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:54.374266 23662 heartbeater.cc:511] Master 127.14.75.62:41391 requested a full tablet report, sending...
I20260810 23:06:54.375335 23191 ts_manager.cc:194] Registered new tserver with Master: f185d287acfe44b9b459a4f94d429b57 (127.14.75.3:34063)
I20260810 23:06:54.375941 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:51443
I20260810 23:06:54.379169 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
Entry for principal oryx/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab.
WARNING: no policy specified for oryx/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
Entry for principal oryx/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab.
WARNING: no policy specified for oryx/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
Entry for principal oryx/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab.
I20260810 23:06:54.460332 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23254
I20260810 23:06:54.467015 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23392
I20260810 23:06:54.473888 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23528
I20260810 23:06:54.481274 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23158
I20260810 23:06:54.488126 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41391
--webserver_interface=127.14.75.62
--webserver_port=39575
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:54.604287 23671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:54.604602 23671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:54.604761 23671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:54.608980 23671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:54.609071 23671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:54.609091 23671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:54.609148 23671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:54.609184 23671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:54.614729 23671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=39575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:54.616460 23671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:54.617913 23671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:54.626803 23679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.626818 23677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.626868 23676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:54.627238 23671 server_base.cc:1034] running on GCE node
I20260810 23:06:54.627830 23671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:54.628989 23671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:54.630201 23671 hybrid_clock.cc:648] HybridClock initialized: now 1786403214630192 us; error 33 us; skew 500 ppm
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:54.633066 23671 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:06:54.634060 23671 webserver.cc:460] Webserver started at http://127.14.75.62:39575/ using document root <none> and password file <none>
I20260810 23:06:54.634658 23671 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:54.634740 23671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:54.638844 23671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:06:54.641070 23686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.642434 23671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:54.642592 23671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "377450e3f22a45d488a175846f3f3bea"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "f3313d669aebcd3a29515d5b6451d249"
server_key_iv: "aec6d5619a69442377866e075543101c"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.643203 23671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:54.662461 23671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:54.665591 23671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:54.665814 23671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:54.673524 23671 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41391
I20260810 23:06:54.673548 23741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41391 every 8 connection(s)
I20260810 23:06:54.674638 23671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:54.675894 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23671
I20260810 23:06:54.676414 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:44083
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:41391
--webserver_port=33259
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:54.679836 23742 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:06:54.681762 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Bootstrap starting.
I20260810 23:06:54.699395 23742 log.cc:824] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:54.706295 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:54.706722 23742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Bootstrap complete.
I20260810 23:06:54.711154 23742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:54.711572 23742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377450e3f22a45d488a175846f3f3bea, State: Initialized, Role: FOLLOWER
I20260810 23:06:54.712064 23742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:54.712189 23742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:54.712276 23742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:54.712376 23742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:06:54.714471 23742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:54.714811 23742 leader_election.cc:302] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 377450e3f22a45d488a175846f3f3bea; no voters:
I20260810 23:06:54.715165 23742 leader_election.cc:288] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:06:54.715260 23746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:06:54.715454 23746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [term 2 LEADER]: Becoming Leader. State: Replica: 377450e3f22a45d488a175846f3f3bea, State: Running, Role: LEADER
I20260810 23:06:54.715758 23746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } }
I20260810 23:06:54.716434 23742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:54.716784 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 377450e3f22a45d488a175846f3f3bea. Latest consensus state: current_term: 2 leader_uuid: "377450e3f22a45d488a175846f3f3bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } } }
I20260810 23:06:54.716921 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:54.717155 23747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "377450e3f22a45d488a175846f3f3bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377450e3f22a45d488a175846f3f3bea" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41391 } } }
I20260810 23:06:54.717274 23747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:54.717522 23750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:54.722114 23750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:54.723178 23750 catalog_manager.cc:1259] Loaded cluster ID: 19b4b57fa0284dfe97413577d379c1b9
I20260810 23:06:54.723258 23750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:54.725669 23750 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:54.726332 23750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 377450e3f22a45d488a175846f3f3bea: Loaded TSK: 0
I20260810 23:06:54.727552 23750 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:06:54.795405 23744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:54.795706 23744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:54.795750 23744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:54.800114 23744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:54.800184 23744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:54.800266 23744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:06:54.805774 23744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.1.keytab
--principal=oryx/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:44083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=33259
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:54.806891 23744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:54.807955 23744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:54.818790 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.818787 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:54.819263 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:54.819430 23744 server_base.cc:1034] running on GCE node
I20260810 23:06:54.819965 23744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:54.820659 23744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:54.821861 23744 hybrid_clock.cc:648] HybridClock initialized: now 1786403214821834 us; error 47 us; skew 500 ppm
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403214, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:54.825129 23744 init.cc:377] Logged in from keytab as oryx/127.14.75.1@KRBTEST.COM (short username oryx)
I20260810 23:06:54.826239 23744 webserver.cc:460] Webserver started at http://127.14.75.1:33259/ using document root <none> and password file <none>
I20260810 23:06:54.826833 23744 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:54.826964 23744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:54.831127 23744 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:06:54.833340 23778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.834734 23744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:06:54.834838 23744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "061d580a864d433db8089b41e851ee9b"
format_stamp: "Formatted at 2026-08-10 23:06:53 on dist-test-slave-srgv"
server_key: "ed6220cf20a3c267f0e495d0e96b9d1b"
server_key_iv: "e33b25377071cac3d329ff68247855f2"
server_key_version: "encryptionkey@0"
I20260810 23:06:54.835440 23744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:54.862280 23744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:54.865625 23744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:54.865846 23744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:54.866483 23744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:54.867375 23744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:54.867425 23744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.867501 23744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:54.867532 23744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:54.877177 23744 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:44083
I20260810 23:06:54.877197 23894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:44083 every 8 connection(s)
I20260810 23:06:54.878331 23744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:54.884333 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23744
I20260810 23:06:54.884865 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:36479
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:41391
--webserver_port=37223
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:54.893517 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.880648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:32839)
0810 23:06:54.881182 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.881247 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:54.882605 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:54.884094 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:54.884103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:54.884638 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:54.885321 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.885347 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.886795 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.886801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.888103 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.888114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.889282 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.889285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.889495 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.890175 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:54.890192 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:54.892636 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":154,"threads_started":1}
W20260810 23:06:54.894026 23895 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:41391 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.75.62:41391: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
W20260810 23:06:54.894747 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:54.880498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:32839 (local address 127.14.75.62:41391)
0810 23:06:54.880977 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:54.880996 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:54.881375 (+ 379us) server_negotiation.cc:407] Connection header received
0810 23:06:54.883162 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:54.883180 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:54.883621 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:54.883911 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:54.885582 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.886633 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.888251 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.889099 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.889321 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.893009 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:32839: BlockingRecv error: recv got EOF from 127.14.75.1:32839 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":96,"threads_started":1}
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:54.900741 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.894530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:34845)
0810 23:06:54.894757 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.894775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:54.894868 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:54.895512 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:54.895515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:54.895662 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:54.895758 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.895763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.896943 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.896945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.897563 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.897572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.898190 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.898193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.898261 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.898815 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:54.898827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:54.900639 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260810 23:06:54.901271 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:54.894621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:34845 (local address 127.14.75.62:41391)
0810 23:06:54.895164 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:54.895181 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:54.895200 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:54.895252 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:54.895257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:54.895367 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:54.895477 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:54.895892 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.896827 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.897673 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.898063 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.898182 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.900813 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:34845: BlockingRecv error: recv got EOF from 127.14.75.1:34845 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":141,"threads_started":1}
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:54 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:54.907065 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.901376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:42897)
0810 23:06:54.901584 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.901597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:54.901690 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:54.901970 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:54.901973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:54.902121 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:54.902195 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.902199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.903224 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.903226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.903731 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:54.903738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.904336 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.904339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:54.904395 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.904992 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:54.905004 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:54.906903 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260810 23:06:54.907353 23895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:06:54.907495 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:54.901462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:42897 (local address 127.14.75.62:41391)
0810 23:06:54.901590 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:54.901594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:54.901668 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:06:54.901772 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:54.901775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:54.901837 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:54.901931 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:54.902298 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.903107 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.903843 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:54.904222 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:54.904329 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:54.907036 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:42897: BlockingRecv error: recv got EOF from 127.14.75.1:42897 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 23:06:55.006634 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:55.006937 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:55.006982 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:55.011260 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:55.011351 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:55.011415 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:06:55.017630 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.2.keytab
--principal=oryx/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:36479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=37223
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:55.018841 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:55.019920 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:55.030808 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:55.030808 23908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:55.031111 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:55.031244 23899 server_base.cc:1034] running on GCE node
I20260810 23:06:55.031699 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:55.032408 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:55.033584 23899 hybrid_clock.cc:648] HybridClock initialized: now 1786403215033564 us; error 33 us; skew 500 ppm
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403215, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:55.036626 23899 init.cc:377] Logged in from keytab as oryx/127.14.75.2@KRBTEST.COM (short username oryx)
I20260810 23:06:55.037611 23899 webserver.cc:460] Webserver started at http://127.14.75.2:37223/ using document root <none> and password file <none>
I20260810 23:06:55.038169 23899 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:55.038245 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:55.042461 23899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:06:55.044564 23915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.045902 23899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:06:55.046032 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "5a827b23b5ea432186cd4bca52929b90"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "050dc992d2b3a1ee84008e6287392423"
server_key_iv: "b10577f3b0bb01290ff2753abd751234"
server_key_version: "encryptionkey@0"
I20260810 23:06:55.046592 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:55.060317 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:55.063958 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:55.064208 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:55.064868 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:55.065833 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:55.065884 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.065964 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:55.066004 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.075121 23899 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36479
I20260810 23:06:55.075150 24031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36479 every 8 connection(s)
I20260810 23:06:55.076190 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:55.083206 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 23899
I20260810 23:06:55.083684 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:34063
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:41391
--webserver_port=36541
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:06:55.088768 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.078472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:45609 (local address 127.14.75.62:41391)
0810 23:06:55.078645 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.078651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.079591 (+ 940us) server_negotiation.cc:407] Connection header received
0810 23:06:55.080883 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.080886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.080943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.081029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.082327 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.082884 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.084237 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.084649 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.084782 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.088581 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:45609: BlockingRecv error: recv got EOF from 127.14.75.2:45609 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:06:55.089262 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.078931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:45609)
0810 23:06:55.079420 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.079465 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.080640 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.081201 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.081209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.081693 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.082183 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.082196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.083014 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.083016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.084093 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.084103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.084828 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.084834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.084984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.085710 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.085728 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.088348 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
W20260810 23:06:55.089913 24033 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:41391 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.75.62:41391: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.095979 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.090406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:39101)
0810 23:06:55.090528 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.090545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.090632 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.090936 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.090939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.091116 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.091220 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.091225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.091972 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.091977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.092538 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.092546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.093008 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.093011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.093102 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.093740 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.093752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.095821 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:06:55.096077 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.090563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:39101 (local address 127.14.75.62:41391)
0810 23:06:55.090732 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.090736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.090748 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:55.090786 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.090788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.090829 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.090918 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.091340 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.091862 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.092674 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.092893 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.093006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.095959 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:39101: BlockingRecv error: recv got EOF from 127.14.75.2:39101 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.102334 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.096649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:52709)
0810 23:06:55.096802 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.096818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.096899 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.097135 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.097138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.097312 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.097438 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.097444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.098144 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.098147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.099053 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.099063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.099523 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.099527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.099618 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.100116 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.100130 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.102149 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 23:06:55.102449 23900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.096774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:52709 (local address 127.14.75.62:41391)
0810 23:06:55.096898 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.096901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.096915 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:55.096989 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.096991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.097026 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.097093 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.097550 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.098039 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.099192 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.099398 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.099474 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.102307 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:52709: BlockingRecv error: recv got EOF from 127.14.75.2:52709 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:06:55.102734 24033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:06:55.203387 24036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:55.203683 24036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:55.203729 24036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:55.207871 24036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:55.207947 24036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:55.208010 24036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:06:55.213649 24036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/krb5kdc/oryx_127.14.75.3.keytab
--principal=oryx/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:34063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=36541
--tserver_master_addrs=127.14.75.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:55.214880 24036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:55.215873 24036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:55.226029 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:55.226054 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:55.226475 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:55.226446 24036 server_base.cc:1034] running on GCE node
I20260810 23:06:55.226945 24036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:55.227567 24036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:55.228767 24036 hybrid_clock.cc:648] HybridClock initialized: now 1786403215228751 us; error 26 us; skew 500 ppm
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403215, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:55.231755 24036 init.cc:377] Logged in from keytab as oryx/127.14.75.3@KRBTEST.COM (short username oryx)
I20260810 23:06:55.232764 24036 webserver.cc:460] Webserver started at http://127.14.75.3:36541/ using document root <none> and password file <none>
I20260810 23:06:55.233366 24036 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:55.233515 24036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:55.237591 24036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:06:55.239877 24052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.241256 24036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:55.241447 24036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "f185d287acfe44b9b459a4f94d429b57"
format_stamp: "Formatted at 2026-08-10 23:06:54 on dist-test-slave-srgv"
server_key: "9328306fe8ea67665bb0ed6ed78ded9b"
server_key_iv: "7f60f57df1d74e83e996a3e2cd06bbd4"
server_key_version: "encryptionkey@0"
I20260810 23:06:55.242045 24036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:55.295588 24036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:55.299054 24036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:55.299301 24036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:55.299948 24036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:55.300881 24036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:55.300959 24036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.301035 24036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:55.301079 24036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:55.310818 24036 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:34063
I20260810 23:06:55.310846 24169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:34063 every 8 connection(s)
I20260810 23:06:55.311884 24036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:55.312211 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24036
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.322716 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.313037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.14.75.62:41391)
0810 23:06:55.313197 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.313203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.313449 (+ 246us) server_negotiation.cc:407] Connection header received
0810 23:06:55.313634 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.313643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.313714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.313798 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.314612 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.315381 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.316653 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.316908 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.317009 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.318698 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:55.318703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:55.318715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:55.318742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:55.320593 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:55.321072 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:55.321077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:55.321083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:55.321212 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:55.321535 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:55.321539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:55.321541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:55.321880 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:55.322092 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:55.322350 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:06:55.322594 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:06:55.324877 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.314559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:50413 (local address 127.14.75.62:41391)
0810 23:06:55.314684 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.314688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.315413 (+ 725us) server_negotiation.cc:407] Connection header received
0810 23:06:55.317207 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.317210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.317258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.317315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.318681 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.319418 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.320563 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.320789 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.320903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.324771 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:50413: BlockingRecv error: recv got EOF from 127.14.75.3:50413 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:55.325856 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.314747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:50413)
0810 23:06:55.315271 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.315313 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.316929 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.317554 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.317562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.318029 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.318535 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.318549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.319551 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.319555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.320430 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.320439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.320948 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.320952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.321124 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.321738 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.321754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.324636 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":146,"threads_started":1}
W20260810 23:06:55.326300 24170 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:41391 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.75.62:41391: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.331674 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.326704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:33191)
0810 23:06:55.326822 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.326838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.326918 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.327163 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.327165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.327303 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.327396 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.327401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.328068 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.328071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.328624 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.328632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.329020 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.329022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.329095 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.329626 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.329637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.331562 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:06:55.331872 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.326795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:33191 (local address 127.14.75.62:41391)
0810 23:06:55.326916 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.326919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.326931 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:55.326983 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.326985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.327049 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.327140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.327490 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.327978 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.328723 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.328927 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.329001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.331724 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:33191: BlockingRecv error: recv got EOF from 127.14.75.3:33191 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.337667 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.332314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:50205)
0810 23:06:55.332453 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.332466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.332570 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.332870 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.332873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.333033 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.333108 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.333113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.333911 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.333916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.334435 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.334443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.334884 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.334887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.334965 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.335412 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.335423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.337511 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 23:06:55.337729 23897 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.332427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:50205 (local address 127.14.75.62:41391)
0810 23:06:55.332636 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.332641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.332655 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:55.332700 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.332702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.332759 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.332853 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.333223 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.333806 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.334556 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.334775 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.334882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.337623 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:50205: BlockingRecv error: recv got EOF from 127.14.75.3:50205 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260810 23:06:55.338075 24170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:55 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:55.913683 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.908146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:43091)
0810 23:06:55.908441 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.908455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:55.908547 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:55.908813 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:55.908815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:55.909003 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:55.909132 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.909137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.909957 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.909959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.910558 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:55.910566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.910971 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.910974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:55.911032 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.911547 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:55.911558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:55.913501 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
W20260810 23:06:55.913820 24175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:55.908286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:43091 (local address 127.14.75.62:41391)
0810 23:06:55.908522 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:55.908526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:55.908540 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:55.908645 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:55.908647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:55.908691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:55.908776 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:55.909236 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.909856 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.910661 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:55.910875 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:55.910948 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:55.913661 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:43091: BlockingRecv error: recv got EOF from 127.14.75.1:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:56.109687 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.103565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:41593)
0810 23:06:56.103869 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.103885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:56.103996 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:56.104348 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:56.104350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:56.104546 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:56.104684 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.104690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.105506 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.105508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.106082 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.106114 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.106662 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.106666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.106777 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.107362 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:56.107375 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:56.109500 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
W20260810 23:06:56.109786 24175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:56.103746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:41593 (local address 127.14.75.62:41391)
0810 23:06:56.103942 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:56.103946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:56.103962 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:56.104154 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:56.104163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:56.104225 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:56.104310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:56.104810 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.105338 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.106255 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.106503 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.106643 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.109653 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:41593: BlockingRecv error: recv got EOF from 127.14.75.2:41593 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:56.344667 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.338749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:40105)
0810 23:06:56.339049 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.339063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:56.339199 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:56.339608 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:56.339614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:56.339824 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:56.339929 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.339935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.340684 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.340687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.341285 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.341293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.341812 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.341815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.341882 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.342453 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:56.342465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:56.344469 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
W20260810 23:06:56.344731 24175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:56.338913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40105 (local address 127.14.75.62:41391)
0810 23:06:56.339084 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:56.339087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:56.339160 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:06:56.339346 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:56.339348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:56.339398 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:56.339471 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:56.340047 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.340577 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.341395 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.341699 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.341790 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.344611 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:40105: BlockingRecv error: recv got EOF from 127.14.75.3:40105 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:56 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:56.921226 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.914679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:50253)
0810 23:06:56.914982 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.914997 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:56.915100 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:56.915431 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:56.915435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:56.915632 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:56.915737 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.915743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.916686 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.916689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.917488 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:56.917500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.918085 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.918088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:56.918159 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.918847 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:56.918860 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:56.921070 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
W20260810 23:06:56.921411 24179 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:56.914867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50253 (local address 127.14.75.62:41391)
0810 23:06:56.915122 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:56.915126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:56.915163 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:56.915202 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:56.915205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:56.915258 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:56.915333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:56.915879 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.916522 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.917705 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:56.917941 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:56.918034 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:56.921260 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:50253: BlockingRecv error: recv got EOF from 127.14.75.1:50253 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":110,"threads_started":1}
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:57.116876 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.110622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:34143)
0810 23:06:57.110992 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.111009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:57.111172 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:57.111459 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:57.111462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:57.111701 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:57.111800 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.111805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.112600 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.112606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.113171 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.113179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.113728 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.113732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.113822 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.114354 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:57.114366 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:57.116653 (+ 2287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
W20260810 23:06:57.117043 24179 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:57.110785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:34143 (local address 127.14.75.62:41391)
0810 23:06:57.110979 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:57.110984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:57.111103 (+ 119us) server_negotiation.cc:407] Connection header received
0810 23:06:57.111285 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:57.111289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:57.111342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:57.111438 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:57.111932 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.112474 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.113298 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.113604 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.113720 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.116889 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:34143: BlockingRecv error: recv got EOF from 127.14.75.2:34143 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:57.351233 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.345552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:49499)
0810 23:06:57.345912 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.345925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:57.346022 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:57.346261 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:57.346263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:57.346451 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:57.346569 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.346574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.347392 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.347394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.348021 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.348030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.348450 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.348452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.348515 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.349006 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:57.349017 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:57.351083 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
W20260810 23:06:57.351328 24179 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:57.345667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:49499 (local address 127.14.75.62:41391)
0810 23:06:57.345823 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:57.345826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:57.345991 (+ 165us) server_negotiation.cc:407] Connection header received
0810 23:06:57.346102 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:57.346104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:57.346153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:57.346224 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:57.346651 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.347292 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.348124 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.348344 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.348425 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.351231 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:49499: BlockingRecv error: recv got EOF from 127.14.75.3:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:57 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:57.928107 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.922267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:42213)
0810 23:06:57.922586 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.922601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:57.922695 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:57.922989 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:57.922991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:57.923181 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:57.923284 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.923290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.924113 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.924116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.924659 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:57.924665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.925146 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.925148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:57.925209 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.925762 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:57.925779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:57.927964 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
W20260810 23:06:57.928292 24183 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:57.922411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:42213 (local address 127.14.75.62:41391)
0810 23:06:57.922669 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:57.922673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:57.922689 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:57.922817 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:57.922820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:57.922865 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:57.922949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:57.923435 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.923997 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.924788 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:57.925029 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:57.925141 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:57.928114 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:42213: BlockingRecv error: recv got EOF from 127.14.75.1:42213 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":61,"threads_started":1}
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:58.123862 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.117821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:60571)
0810 23:06:58.118099 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.118113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:58.118223 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:58.118502 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:58.118505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:58.118756 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:58.118860 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.118867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.119814 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.119819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.120507 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.120519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.121109 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.121112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.121177 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.121782 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:58.121793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:58.123741 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
W20260810 23:06:58.124110 24183 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:58.117911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:60571 (local address 127.14.75.62:41391)
0810 23:06:58.118118 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:58.118122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:58.118171 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:06:58.118331 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:58.118333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:58.118383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:58.118458 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:58.118984 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.119694 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.120653 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.120985 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.121099 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.123902 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:60571: BlockingRecv error: recv got EOF from 127.14.75.2:60571 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:58.357935 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.352337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:44755)
0810 23:06:58.352658 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.352679 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:58.352817 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:58.353132 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:58.353135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:58.353345 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:58.353514 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.353521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.354264 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.354267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.354818 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.354826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.355254 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.355256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.355315 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.355870 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:58.355881 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:58.357773 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
W20260810 23:06:58.358115 24183 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:58.352432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:44755 (local address 127.14.75.62:41391)
0810 23:06:58.352681 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:58.352688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:58.352774 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:06:58.352935 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:58.352940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:58.353001 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:58.353115 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:58.353620 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.354151 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.354921 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.355145 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.355221 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.357941 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:44755: BlockingRecv error: recv got EOF from 127.14.75.3:44755 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:58 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:58.937901 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.929058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:39339)
0810 23:06:58.929395 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.929429 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:58.929525 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:58.929886 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:58.929889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:58.930080 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:58.930176 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.930181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.930946 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.930948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.931498 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:58.931505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.932009 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.932011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:58.932071 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.935683 (+ 3612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:58.935696 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:58.937777 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
W20260810 23:06:58.938104 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:58.929177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:39339 (local address 127.14.75.62:41391)
0810 23:06:58.929650 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:58.929654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:58.929671 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:58.929716 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:58.929718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:58.929761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:58.929851 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:58.930307 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.930846 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.931639 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:58.931889 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:58.931985 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:58.937943 (+ 5958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:39339: BlockingRecv error: recv got EOF from 127.14.75.1:39339 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":68,"threads_started":1}
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:59.130699 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.124829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:52925)
0810 23:06:59.125137 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.125150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:59.125280 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:59.125644 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:59.125647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:59.125907 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:59.126041 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.126047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.126813 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.126818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.127385 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.127392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.127889 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.127893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.127980 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.128493 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:59.128505 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:59.130552 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
W20260810 23:06:59.130897 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:59.124911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:52925 (local address 127.14.75.62:41391)
0810 23:06:59.125071 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:59.125075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:59.125224 (+ 149us) server_negotiation.cc:407] Connection header received
0810 23:06:59.125448 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:59.125455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:59.125516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:59.125614 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:59.126166 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.126700 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.127517 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.127758 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.127886 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.130731 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:52925: BlockingRecv error: recv got EOF from 127.14.75.2:52925 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:59.364530 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.358846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:40577)
0810 23:06:59.359125 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.359139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:59.359249 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:59.359499 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:59.359501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:59.359692 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:59.359792 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.359797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.360570 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.360573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.361196 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.361205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.361722 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.361724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.361796 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.362331 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:59.362344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:59.364330 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
W20260810 23:06:59.364729 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:59.358940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40577 (local address 127.14.75.62:41391)
0810 23:06:59.359111 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:59.359115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:59.359193 (+ 78us) server_negotiation.cc:407] Connection header received
0810 23:06:59.359321 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:59.359323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:59.359370 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:59.359462 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:59.359904 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.360432 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.361327 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.361604 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.361691 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.364601 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:40577: BlockingRecv error: recv got EOF from 127.14.75.3:40577 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:06:59 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:06:59.944895 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.938897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:54745)
0810 23:06:59.939220 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.939234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:59.939326 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:59.939604 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:59.939607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:59.939827 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:59.939921 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.939927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.940731 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.940736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.941315 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:59.941323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.941827 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.941830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:59.941913 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.942503 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:59.942515 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:59.944674 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":139,"threads_started":1}
W20260810 23:06:59.945153 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:59.939098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:54745 (local address 127.14.75.62:41391)
0810 23:06:59.939338 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:59.939342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:59.939357 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:59.939400 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:59.939402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:59.939443 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:59.939583 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:59.940050 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.940619 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.941468 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:59.941707 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:59.941820 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:59.944922 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:54745: BlockingRecv error: recv got EOF from 127.14.75.1:54745 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:00.137647 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.131692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:39281)
0810 23:07:00.132015 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.132032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:00.132145 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:00.132409 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:00.132411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:00.132628 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:00.132729 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.132735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.133635 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.133638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.134214 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.134223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.134689 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.134692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.134755 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.135294 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:00.135306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:00.137456 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":130,"threads_started":1}
W20260810 23:07:00.137867 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:00.131878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:39281 (local address 127.14.75.62:41391)
0810 23:07:00.132018 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:00.132021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:00.132160 (+ 139us) server_negotiation.cc:407] Connection header received
0810 23:07:00.132229 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:00.132232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:00.132282 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:00.132365 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:00.132850 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.133519 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.134348 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.134577 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.134660 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.137604 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:39281: BlockingRecv error: recv got EOF from 127.14.75.2:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:00.371786 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.365707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:47897)
0810 23:07:00.366004 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.366017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:00.366124 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:00.366387 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:00.366389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:00.366584 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:00.366679 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.366683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.367511 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.367513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.368200 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.368210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.368782 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.368786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.368875 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.369573 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:00.369586 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:00.371658 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
W20260810 23:07:00.371996 24191 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:00.365811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:47897 (local address 127.14.75.62:41391)
0810 23:07:00.365998 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:00.366002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:00.366111 (+ 109us) server_negotiation.cc:407] Connection header received
0810 23:07:00.366210 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:00.366213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:00.366260 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:00.366347 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:00.366803 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.367395 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.368407 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.368646 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.368767 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.371821 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:47897: BlockingRecv error: recv got EOF from 127.14.75.3:47897 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:00 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:00.952745 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.946224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:55847)
0810 23:07:00.946519 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.946540 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:00.946664 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:00.947066 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:00.947069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:00.947284 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:00.947385 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.947391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.948447 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.948452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.949173 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:00.949185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.949771 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.949775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:00.949865 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.950500 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:00.950512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:00.952613 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
W20260810 23:07:00.952965 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:00.946356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:55847 (local address 127.14.75.62:41391)
0810 23:07:00.946730 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:00.946733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:00.946750 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:07:00.946796 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:00.946799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:00.946843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:00.947021 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:00.947525 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.948304 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.949319 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:00.949611 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:00.949698 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:00.952791 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:55847: BlockingRecv error: recv got EOF from 127.14.75.1:55847 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":137,"threads_started":1}
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:01.144999 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.138592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:43431)
0810 23:07:01.138983 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.139001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:01.139128 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:01.139467 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:01.139471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:01.139758 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:01.139881 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.139888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.140718 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.140723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.141295 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.141304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.141829 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.141833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.141930 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.142589 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:01.142604 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:01.144824 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
W20260810 23:07:01.145079 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:01.138789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:43431 (local address 127.14.75.62:41391)
0810 23:07:01.138966 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:01.138971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:01.139065 (+ 94us) server_negotiation.cc:407] Connection header received
0810 23:07:01.139265 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:01.139268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:01.139327 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:01.139419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:01.140013 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.140593 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.141461 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.141705 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.141821 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.144954 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:43431: BlockingRecv error: recv got EOF from 127.14.75.2:43431 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:01.378638 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:51381)
0810 23:07:01.372976 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.372990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:01.373106 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:01.373438 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:01.373441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:01.373745 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:01.373874 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.373884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.374669 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.374671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.375251 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.375259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.375757 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.375760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.375840 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.376383 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:01.376396 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:01.378504 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
W20260810 23:07:01.378801 24195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:01.372765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:51381 (local address 127.14.75.62:41391)
0810 23:07:01.372919 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:01.372924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:01.373075 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:07:01.373235 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:01.373237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:01.373287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:01.373359 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:01.374000 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.374541 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.375385 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.375640 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.375733 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.378671 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:51381: BlockingRecv error: recv got EOF from 127.14.75.3:51381 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:01 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:01.959647 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.953615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:46661)
0810 23:07:01.953961 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.953975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:01.954071 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:01.954489 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:01.954491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:01.954692 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:01.954793 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.954799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.955605 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.955607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.956179 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:01.956188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.956634 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.956636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:01.956701 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.957207 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:01.957219 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:01.959462 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
W20260810 23:07:01.959757 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:01.953673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:46661 (local address 127.14.75.62:41391)
0810 23:07:01.954272 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:01.954276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:01.954288 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:07:01.954323 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:01.954325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:01.954363 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:01.954436 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:01.954906 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.955450 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.956291 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:01.956513 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:01.956598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:01.959615 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:46661: BlockingRecv error: recv got EOF from 127.14.75.1:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":88,"threads_started":1}
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:02.151760 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.145887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:59239)
0810 23:07:02.146321 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.146335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:02.146438 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:02.146767 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:02.146770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:02.146982 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:02.147080 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.147086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.147905 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.147913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.148475 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.148483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.148908 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.148911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.148975 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.149592 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:02.149604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:02.151594 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":93,"threads_started":1}
W20260810 23:07:02.151877 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:02.146011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:59239 (local address 127.14.75.62:41391)
0810 23:07:02.146216 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:02.146221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:02.146416 (+ 195us) server_negotiation.cc:407] Connection header received
0810 23:07:02.146522 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:02.146524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:02.146570 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:02.146649 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:02.147254 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.147795 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.148579 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.148802 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.148886 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.151752 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:59239: BlockingRecv error: recv got EOF from 127.14.75.2:59239 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:02.385540 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.379656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:40215)
0810 23:07:02.379999 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.380013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:02.380122 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:02.380383 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:02.380385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:02.380584 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:02.380680 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.380686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.381488 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.381491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.382161 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.382173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.382679 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.382682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.382773 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.383381 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:02.383394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:02.385338 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
W20260810 23:07:02.385612 24199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:02.379734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40215 (local address 127.14.75.62:41391)
0810 23:07:02.379903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:02.379907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:02.380099 (+ 192us) server_negotiation.cc:407] Connection header received
0810 23:07:02.380210 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:02.380212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:02.380261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:02.380344 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:02.380808 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.381322 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.382295 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.382550 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.382667 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.385500 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:40215: BlockingRecv error: recv got EOF from 127.14.75.3:40215 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:02 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:02.966689 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.960637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:60575)
0810 23:07:02.960995 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.961008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:02.961105 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:02.961377 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:02.961379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:02.961605 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:02.961701 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.961706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.962441 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.962443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.963255 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:02.963266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.963698 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.963701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:02.963764 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.964382 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:02.964394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:02.966510 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":181,"threads_started":1}
W20260810 23:07:02.966838 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:02.960891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:60575 (local address 127.14.75.62:41391)
0810 23:07:02.961132 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:02.961135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:02.961167 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:07:02.961204 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:02.961206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:02.961257 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:02.961340 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:02.961806 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.962338 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.963372 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:02.963592 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:02.963668 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:02.966683 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:60575: BlockingRecv error: recv got EOF from 127.14.75.1:60575 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:03.158910 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.152714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:55067)
0810 23:07:03.153037 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.153050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:03.153155 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:03.153470 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:03.153472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:03.153665 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:03.153767 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.153773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.154658 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.154661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.155231 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.155240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.155692 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.155695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.155768 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.156384 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:03.156397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:03.158732 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":152,"threads_started":1}
W20260810 23:07:03.158975 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:03.152871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:55067 (local address 127.14.75.62:41391)
0810 23:07:03.153031 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:03.153035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:03.153127 (+ 92us) server_negotiation.cc:407] Connection header received
0810 23:07:03.153244 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:03.153246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:03.153292 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:03.153375 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:03.153882 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.154552 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.155342 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.155560 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.155638 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.158868 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:55067: BlockingRecv error: recv got EOF from 127.14.75.2:55067 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:03.392078 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.386428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:56163)
0810 23:07:03.386760 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.386774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:03.386870 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:03.387106 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:03.387109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:03.387294 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:03.387387 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.387392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.388185 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.388188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.388749 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.388759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.389207 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.389210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.389271 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.389880 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:03.389892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:03.391957 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
W20260810 23:07:03.392288 24203 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:03.386477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:56163 (local address 127.14.75.62:41391)
0810 23:07:03.386607 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:03.386611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:03.386868 (+ 257us) server_negotiation.cc:407] Connection header received
0810 23:07:03.386934 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:03.386936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:03.386994 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:03.387066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:03.387483 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.388021 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.388867 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.389087 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.389171 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.392120 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:56163: BlockingRecv error: recv got EOF from 127.14.75.3:56163 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:03 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:03.976759 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.967509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:45309)
0810 23:07:03.967813 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.967833 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:03.967959 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:03.968247 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:03.968250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:03.968456 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:03.968559 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.968564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.969335 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.969338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.970063 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:03.970074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.970716 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.970719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:03.970807 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.974489 (+ 3682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:03.974503 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:03.976591 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
W20260810 23:07:03.976899 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:03.967681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:45309 (local address 127.14.75.62:41391)
0810 23:07:03.967949 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:03.967952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:03.967969 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:07:03.968049 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:03.968052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:03.968096 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:03.968178 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:03.968672 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.969212 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.970235 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:03.970602 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:03.970711 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:03.976763 (+ 6052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:45309: BlockingRecv error: recv got EOF from 127.14.75.1:45309 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:04.165876 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.159895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:40585)
0810 23:07:04.160168 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.160186 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:04.160307 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:04.160571 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:04.160573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:04.160777 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:04.160877 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.160883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.161895 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.161897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.162618 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.162629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.163044 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.163047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.163108 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.163722 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:04.163734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:04.165758 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
W20260810 23:07:04.166172 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:04.159982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:40585 (local address 127.14.75.62:41391)
0810 23:07:04.160192 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:04.160196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:04.160280 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:07:04.160400 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:04.160402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:04.160452 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:04.160539 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:04.161013 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.161792 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.162732 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.162947 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.163030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.165910 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:40585: BlockingRecv error: recv got EOF from 127.14.75.2:40585 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:04.398823 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.392890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:34753)
0810 23:07:04.393262 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.393275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:04.393434 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:04.393725 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:04.393728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:04.393944 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:04.394039 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.394044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.394833 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.394838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.395401 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.395409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.395869 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.395872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.395957 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.396518 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:04.396530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:04.398646 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
W20260810 23:07:04.398926 24207 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:04.393026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:34753 (local address 127.14.75.62:41391)
0810 23:07:04.393198 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:04.393205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:04.393355 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:07:04.393562 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:04.393565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:04.393610 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:04.393700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:04.394193 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.394724 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.395522 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.395752 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.395863 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.398803 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:34753: BlockingRecv error: recv got EOF from 127.14.75.3:34753 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:04 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:04.983366 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.977795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:40435)
0810 23:07:04.978143 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.978157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:04.978251 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:04.978506 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:04.978508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:04.978697 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:04.978795 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.978800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.979514 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.979516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.980082 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:04.980090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.980501 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.980504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:04.980568 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.981145 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:04.981157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:04.983186 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":73,"threads_started":1}
W20260810 23:07:04.983444 24211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:04.977874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:40435 (local address 127.14.75.62:41391)
0810 23:07:04.978176 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:04.978180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:04.978212 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:07:04.978341 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:04.978343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:04.978385 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:04.978466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:04.978894 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.979415 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.980189 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:04.980402 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:04.980476 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:04.983315 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:40435: BlockingRecv error: recv got EOF from 127.14.75.1:40435 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:05.172406 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.166771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:49355)
0810 23:07:05.167066 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.167083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:05.167184 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:05.167420 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:05.167422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:05.167625 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:05.167722 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.167728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.168490 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.168494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.169059 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.169068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.169619 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.169622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.169690 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.170205 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:05.170216 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:05.172225 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
W20260810 23:07:05.172483 24211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:05.166823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:49355 (local address 127.14.75.62:41391)
0810 23:07:05.166984 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:05.166989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:05.167191 (+ 202us) server_negotiation.cc:407] Connection header received
0810 23:07:05.167261 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:05.167263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:05.167310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:05.167384 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:05.167834 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.168360 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.169242 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.169504 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.169591 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.172368 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:49355: BlockingRecv error: recv got EOF from 127.14.75.2:49355 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:05.405629 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.399749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:53873)
0810 23:07:05.400088 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.400103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:05.400206 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:05.400471 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:05.400474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:05.400670 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:05.400780 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.400786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.401704 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.401707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.402307 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.402316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.402797 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.402799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.402862 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.403377 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:05.403389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:05.405413 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
W20260810 23:07:05.405869 24211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:05.399980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53873 (local address 127.14.75.62:41391)
0810 23:07:05.400106 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:05.400109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:05.400182 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:07:05.400307 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:05.400309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:05.400354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:05.400425 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:05.400926 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.401521 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.402446 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.402688 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.402777 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.405593 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:53873: BlockingRecv error: recv got EOF from 127.14.75.3:53873 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:05 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:05.989833 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.984412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:39233)
0810 23:07:05.984710 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.984730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:05.984830 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:05.985054 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:05.985056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:05.985259 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:05.985357 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.985363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.986163 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.986165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.986715 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:05.986723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.987173 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.987175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:05.987240 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.987769 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:05.987781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:05.989691 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
W20260810 23:07:05.989995 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:05.984484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:39233 (local address 127.14.75.62:41391)
0810 23:07:05.984736 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:05.984739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:05.984784 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:07:05.984898 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:05.984900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:05.984941 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:05.985016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:05.985496 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.986040 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.986819 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:05.987045 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:05.987136 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:05.989851 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:39233: BlockingRecv error: recv got EOF from 127.14.75.1:39233 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:06.179207 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:06.173362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:36957)
0810 23:07:06.173711 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:06.173729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:06.173872 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:06.174161 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:06.174164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:06.174391 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:06.174484 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.174490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.175259 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.175264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.175829 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.175838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.176325 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.176329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.176420 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.176968 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:06.176980 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:06.179035 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
W20260810 23:07:06.179316 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:06.173398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:36957 (local address 127.14.75.62:41391)
0810 23:07:06.173579 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:06.173584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:06.173822 (+ 238us) server_negotiation.cc:407] Connection header received
0810 23:07:06.173982 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:06.173986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:06.174042 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:06.174157 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:06.174606 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.175140 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.175956 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.176195 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.176306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.179200 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:36957: BlockingRecv error: recv got EOF from 127.14.75.2:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:06.412006 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:06.406644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:54615)
0810 23:07:06.406961 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:06.406975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:06.407096 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:06.407342 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:06.407344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:06.407535 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:06.407642 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.407651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.408358 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.408360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.408909 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.408917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.409326 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.409328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.409388 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.409958 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:06.409969 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:06.411856 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
W20260810 23:07:06.412145 24215 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:06.406733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:54615 (local address 127.14.75.62:41391)
0810 23:07:06.406890 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:06.406897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:06.407052 (+ 155us) server_negotiation.cc:407] Connection header received
0810 23:07:06.407181 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:06.407184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:06.407230 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:06.407301 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:06.407749 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.408263 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.409013 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.409225 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.409298 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.412015 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:54615: BlockingRecv error: recv got EOF from 127.14.75.3:54615 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:06 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:06.996990 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:06.990768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:54303)
0810 23:07:06.991086 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:06.991101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:06.991198 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:06.991457 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:06.991459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:06.991654 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:06.991755 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.991760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.992544 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.992548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.993219 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:06.993230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.993891 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.993894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:06.993961 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.994655 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:06.994669 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:06.996848 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":122,"threads_started":1}
W20260810 23:07:06.997344 24221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:06.990924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:54303 (local address 127.14.75.62:41391)
0810 23:07:06.991180 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:06.991184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:06.991201 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:07:06.991275 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:06.991278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:06.991322 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:06.991415 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:06.991899 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.992437 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.993333 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.993740 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.993870 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.997028 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:54303: BlockingRecv error: recv got EOF from 127.14.75.1:54303 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:07.186230 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.180168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:35097)
0810 23:07:07.180566 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.180583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.180737 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.181038 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.181041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.181274 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.181371 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.181376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.182185 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.182189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.182762 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.182770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.183357 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.183361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.183450 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.184157 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:07.184169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.186110 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":119,"threads_started":1}
W20260810 23:07:07.186342 24221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:07.180345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:35097 (local address 127.14.75.62:41391)
0810 23:07:07.180548 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.180553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.180682 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:07:07.180857 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.180860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.180916 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.181012 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.181531 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.182069 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.182891 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.183223 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.183351 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.186223 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:35097: BlockingRecv error: recv got EOF from 127.14.75.2:35097 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:07.419142 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.412924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:35691)
0810 23:07:07.413313 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.413329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.413499 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.413795 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.413798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.414025 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.414120 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.414125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.414909 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.414915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.415481 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.415490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.415969 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.415972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.416090 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.416856 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:07.416868 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.419010 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
W20260810 23:07:07.419297 24221 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:07.413078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:35691 (local address 127.14.75.62:41391)
0810 23:07:07.413262 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.413267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.413435 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:07:07.413622 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.413625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.413684 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.413772 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.414247 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.414784 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.415603 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.415847 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.415963 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.419171 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:35691: BlockingRecv error: recv got EOF from 127.14.75.3:35691 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:07 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:08.004117 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.997984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:50217)
0810 23:07:07.998306 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.998325 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.998447 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.998751 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.998754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.998982 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.999118 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.999124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.999901 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.999906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.000479 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.000487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.000984 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.000988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.001073 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.001692 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:08.001705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:08.003903 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":119,"threads_started":1}
W20260810 23:07:08.004235 24225 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:07.998041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50217 (local address 127.14.75.62:41391)
0810 23:07:07.998308 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.998314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.998387 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:07:07.998561 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.998565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.998624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.998725 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.999236 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.999784 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.000627 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.000865 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.000978 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.004089 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:50217: BlockingRecv error: recv got EOF from 127.14.75.1:50217 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
Aug 10 23:07:08 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:08 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:08.193156 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.187161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:55807)
0810 23:07:08.187456 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.187470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.187554 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.187814 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.187816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.188009 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:08.188105 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.188111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.188841 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.188843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.189511 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.189523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.189977 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.189979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.190043 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.190663 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:08.190682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:08.192996 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260810 23:07:08.193368 24225 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:08.187334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:55807 (local address 127.14.75.62:41391)
0810 23:07:08.187494 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.187498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.187541 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:07:08.187650 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.187653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.187700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.187771 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:08.188211 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.188738 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.189641 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.189870 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.189955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.193187 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:55807: BlockingRecv error: recv got EOF from 127.14.75.2:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 23:07:08 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:08 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:08.425792 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.420065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:38263)
0810 23:07:08.420447 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.420464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.420595 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.420910 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.420913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.421127 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:08.421237 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.421243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.421974 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.421976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.422554 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.422562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.422966 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.422968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.423029 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.423649 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:08.423661 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:08.425644 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
W20260810 23:07:08.425926 24225 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:08.420119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:38263 (local address 127.14.75.62:41391)
0810 23:07:08.420260 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.420264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.420560 (+ 296us) server_negotiation.cc:407] Connection header received
0810 23:07:08.420690 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.420693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.420757 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.420857 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:08.421344 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.421882 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.422657 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.422868 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.422942 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.425779 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:38263: BlockingRecv error: recv got EOF from 127.14.75.3:38263 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:09.014550 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.005228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:52247)
0810 23:07:09.005576 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:09.005597 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:09.005730 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:09.005988 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:09.005990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:09.006273 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:09.006446 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.006453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.007383 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.007387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.008069 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.008080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.008503 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.008506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.008568 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.012339 (+ 3771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:09.012353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:09.014396 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
W20260810 23:07:09.014781 24229 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:09.005401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:52247 (local address 127.14.75.62:41391)
0810 23:07:09.005759 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:09.005762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:09.005778 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:07:09.005821 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:09.005824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:09.005867 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:09.005948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:09.006547 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.007245 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.008181 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.008404 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.008486 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.014616 (+ 6130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:52247: BlockingRecv error: recv got EOF from 127.14.75.1:52247 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":95,"threads_started":1}
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:09.200601 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.194111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:54191)
0810 23:07:09.194413 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:09.194427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:09.194538 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:09.194971 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:09.194974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:09.195261 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:09.195390 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.195400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.196291 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.196294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.197032 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.197043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.197676 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.197679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.197746 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.198375 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:09.198388 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:09.200484 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
W20260810 23:07:09.200776 24229 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:09.194206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:54191 (local address 127.14.75.62:41391)
0810 23:07:09.194406 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:09.194410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:09.194504 (+ 94us) server_negotiation.cc:407] Connection header received
0810 23:07:09.194732 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:09.194736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:09.194800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:09.194903 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:09.195533 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.196189 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.197151 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.197377 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.197507 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.200634 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:54191: BlockingRecv error: recv got EOF from 127.14.75.2:54191 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:09 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.3@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:09.431893 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.426666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.3:59251)
0810 23:07:09.426983 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:09.426996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:09.427096 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:09.427321 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:09.427324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:09.427518 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:09.427612 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.427617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.428298 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.428300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.428859 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.428867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.429277 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.429279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.429338 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.429861 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:09.429873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:09.431705 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
W20260810 23:07:09.431965 24229 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:09.426768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:59251 (local address 127.14.75.62:41391)
0810 23:07:09.426901 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:09.426905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:09.427068 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:07:09.427165 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:09.427167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:09.427213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:09.427279 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:09.427709 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.428209 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.428966 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.429181 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.429259 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.431836 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.3:59251: BlockingRecv error: recv got EOF from 127.14.75.3:59251 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:07:10 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:10 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.1@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:10.021263 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.015533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.1:57683)
0810 23:07:10.015830 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.015844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.015928 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.016206 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.016208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.016400 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.016518 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.016524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.017505 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.017513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.018075 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.018084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.018513 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.018517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.018645 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.019270 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:10.019282 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.021142 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
W20260810 23:07:10.021504 24233 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:10.015676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:57683 (local address 127.14.75.62:41391)
0810 23:07:10.015922 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.015925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.015939 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:07:10.016000 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.016002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.016071 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.016167 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.016667 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.017338 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.018184 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.018413 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.018495 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.021317 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.1:57683: BlockingRecv error: recv got EOF from 127.14.75.1:57683 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
Aug 10 23:07:10 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:07:10 dist-test-slave-srgv krb5kdc[23145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.75.2@KRBTEST.COM for oryx/127.14.75.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:07:10.207500 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.201740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41391 (local address 127.14.75.2:48003)
0810 23:07:10.202034 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.202048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.202145 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.202395 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.202397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.202593 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.202696 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.202702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.203427 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.203429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.203976 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.203983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.204422 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.204425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.204489 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.205031 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:10.205042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.207381 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.75.62:41391: Server oryx/127.14.75.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":132,"threads_started":1}
W20260810 23:07:10.207696 24233 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:10.201832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:48003 (local address 127.14.75.62:41391)
0810 23:07:10.201987 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.201993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.202101 (+ 108us) server_negotiation.cc:407] Connection header received
0810 23:07:10.202241 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.202243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.202289 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.202353 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.202805 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.203326 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.204098 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.204316 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.204395 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.207528 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.75.2:48003: BlockingRecv error: recv got EOF from 127.14.75.2:48003 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:07:10.313203 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23744
I20260810 23:07:10.320562 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23899
I20260810 23:07:10.326776 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24036
I20260810 23:07:10.334149 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 23671
2026-08-10T23:07:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18157 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:10Z Disabled control of system clock
I20260810 23:07:10.366895 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42111
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:07:10.481026 24238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.481353 24238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.481391 24238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.485600 24238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:10.485687 24238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.485708 24238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.485752 24238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:10.485769 24238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:10.490767 24238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:10.491920 24238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.493374 24238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:10.502386 24238 server_base.cc:1034] running on GCE node
W20260810 23:07:10.502311 24243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.502353 24244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.502311 24246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.503320 24238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.504521 24238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.505713 24238 hybrid_clock.cc:648] HybridClock initialized: now 1786403230505685 us; error 44 us; skew 500 ppm
I20260810 23:07:10.507751 24238 webserver.cc:460] Webserver started at http://127.14.75.62:42125/ using document root <none> and password file <none>
I20260810 23:07:10.508378 24238 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.508478 24238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.508726 24238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.510918 24238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "bb71de9bfe464f86adfc748beba66ddf"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "799e62bf0a73957c9b19ec50e9f4d350"
server_key_iv: "c1fd16b4e492ca531321d40d22c79515"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.511514 24238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "bb71de9bfe464f86adfc748beba66ddf"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "799e62bf0a73957c9b19ec50e9f4d350"
server_key_iv: "c1fd16b4e492ca531321d40d22c79515"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.515487 24238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:07:10.518137 24252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.519479 24238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:07:10.519641 24238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "bb71de9bfe464f86adfc748beba66ddf"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "799e62bf0a73957c9b19ec50e9f4d350"
server_key_iv: "c1fd16b4e492ca531321d40d22c79515"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.519779 24238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.552521 24238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.553324 24238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.553574 24238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.563107 24238 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42111
I20260810 23:07:10.563186 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42111 every 8 connection(s)
I20260810 23:07:10.564234 24238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:10.567710 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:10.573768 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24238
I20260810 23:07:10.573899 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:07:10.574076 24306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf: Bootstrap starting.
I20260810 23:07:10.574239 14636 external_mini_cluster.cc:1428] Setting key 53b448952059bf56b133c67ac3def97a
I20260810 23:07:10.577211 24306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:10.578055 24306 log.cc:824] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:10.580092 24306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf: No bootstrap required, opened a new log
I20260810 23:07:10.583854 24306 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } }
I20260810 23:07:10.584147 24306 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:10.584239 24306 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb71de9bfe464f86adfc748beba66ddf, State: Initialized, Role: FOLLOWER
I20260810 23:07:10.584640 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.575886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.14.75.62:42111)
0810 23:07:10.576758 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.576773 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.576812 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:07:10.577672 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.577695 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.578133 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.578566 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.579032 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.580023 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.581034 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.581600 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.581792 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.582721 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.582726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.582739 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.583153 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.583218 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.583238 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.583351 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.583474 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.583771 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":186,"threads_started":1}
I20260810 23:07:10.584766 24306 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } }
I20260810 23:07:10.584990 24306 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:10.585067 24306 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:10.585168 24306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:10.586324 24306 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } }
I20260810 23:07:10.586680 24306 leader_election.cc:302] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb71de9bfe464f86adfc748beba66ddf; no voters:
I20260810 23:07:10.587054 24306 leader_election.cc:288] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:10.587209 24313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:10.587507 24313 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [term 1 LEADER]: Becoming Leader. State: Replica: bb71de9bfe464f86adfc748beba66ddf, State: Running, Role: LEADER
I20260810 23:07:10.587842 24313 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } }
I20260810 23:07:10.588366 24306 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:10.589524 24315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb71de9bfe464f86adfc748beba66ddf. Latest consensus state: current_term: 1 leader_uuid: "bb71de9bfe464f86adfc748beba66ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } } }
I20260810 23:07:10.589684 24315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:10.589994 24314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb71de9bfe464f86adfc748beba66ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb71de9bfe464f86adfc748beba66ddf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42111 } } }
I20260810 23:07:10.590092 24314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:10.590109 24322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:10.593336 24322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:10.599807 24322 catalog_manager.cc:1347] Generated new cluster ID: be18ebcdcb3f42d3b9b4c0f3467011c8
I20260810 23:07:10.599910 24322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:10.631938 24322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:10.632959 24322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:10.637158 24322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb71de9bfe464f86adfc748beba66ddf: Generated new TSK 0
I20260810 23:07:10.638077 24322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:10.643667 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--builtin_ntp_servers=127.14.75.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:07:10.768088 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.768404 24332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.768443 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.772683 24332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.772753 24332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.772809 24332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:07:10.777660 24332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:10.778798 24332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.780278 24332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.790553 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.790562 24338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.790565 24340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.790832 24332 server_base.cc:1034] running on GCE node
I20260810 23:07:10.791457 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.792116 24332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.793321 24332 hybrid_clock.cc:648] HybridClock initialized: now 1786403230793304 us; error 29 us; skew 500 ppm
I20260810 23:07:10.795267 24332 webserver.cc:460] Webserver started at http://127.14.75.1:44329/ using document root <none> and password file <none>
I20260810 23:07:10.795910 24332 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.796018 24332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.796272 24332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.798566 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "b313811730404b60bb4a85302200c180"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d137b977e745bdc6acde54caabb95633"
server_key_iv: "b2893b49cb6ff4223c6af9bd5000b328"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.799196 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "b313811730404b60bb4a85302200c180"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d137b977e745bdc6acde54caabb95633"
server_key_iv: "b2893b49cb6ff4223c6af9bd5000b328"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.803160 24332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:07:10.806116 24346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.807421 24332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:10.807570 24332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "b313811730404b60bb4a85302200c180"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d137b977e745bdc6acde54caabb95633"
server_key_iv: "b2893b49cb6ff4223c6af9bd5000b328"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.807705 24332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.826649 24332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.827415 24332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.827637 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.828272 24332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.829269 24332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.829346 24332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.829416 24332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.829504 24332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.838409 24332 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:37069
I20260810 23:07:10.838430 24464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:37069 every 8 connection(s)
I20260810 23:07:10.839373 24332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:10.841353 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24332
I20260810 23:07:10.841492 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:10.841792 14636 external_mini_cluster.cc:1428] Setting key fb1d935dcd6f97ec86f47ee081937c19
I20260810 23:07:10.843859 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--builtin_ntp_servers=127.14.75.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:07:10.850001 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.841357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50283 (local address 127.14.75.62:42111)
0810 23:07:10.841527 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.841531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.842758 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 23:07:10.844173 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.844178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.844248 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.844345 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.845273 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.845928 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.847735 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.847948 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.848025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.849503 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.849506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.849508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.849522 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.849542 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.849550 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.849610 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.849825 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.849882 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:07:10.851442 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.841819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42111 (local address 127.14.75.1:50283)
0810 23:07:10.842580 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.842635 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.843916 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.844509 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.844516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.844600 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.845108 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.845120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.846070 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.846076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.847602 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.847617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.848123 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.848130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.848383 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.849093 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:10.849097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.849163 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:10.849273 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.849644 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.849658 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.849666 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.849825 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.850211 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260810 23:07:10.852842 24465 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42111
I20260810 23:07:10.853102 24465 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.853725 24465 heartbeater.cc:511] Master 127.14.75.62:42111 requested a full tablet report, sending...
I20260810 23:07:10.855273 24269 ts_manager.cc:194] Registered new tserver with Master: b313811730404b60bb4a85302200c180 (127.14.75.1:37069)
I20260810 23:07:10.856439 24269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.1:50283
W20260810 23:07:10.962594 24468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.962949 24468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.963040 24468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.967386 24468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.967481 24468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.967581 24468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:07:10.972527 24468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:10.973763 24468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.975211 24468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.985448 24476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.985455 24473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.985496 24474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.985869 24468 server_base.cc:1034] running on GCE node
I20260810 23:07:10.986392 24468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.987092 24468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.988363 24468 hybrid_clock.cc:648] HybridClock initialized: now 1786403230988325 us; error 49 us; skew 500 ppm
I20260810 23:07:10.995234 24468 webserver.cc:460] Webserver started at http://127.14.75.2:40265/ using document root <none> and password file <none>
I20260810 23:07:10.996032 24468 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.996141 24468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.996410 24468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.998749 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "62fd5fb0be2147b78e45f5707fff55c6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d6fec3945c339e8d65e51aff412ea5f2"
server_key_iv: "41f29d10394303916eb4d2aae66476fb"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.999382 24468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "62fd5fb0be2147b78e45f5707fff55c6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d6fec3945c339e8d65e51aff412ea5f2"
server_key_iv: "41f29d10394303916eb4d2aae66476fb"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.003530 24468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:07:11.006412 24482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.007721 24468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:07:11.007870 24468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "62fd5fb0be2147b78e45f5707fff55c6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-srgv"
server_key: "d6fec3945c339e8d65e51aff412ea5f2"
server_key_iv: "41f29d10394303916eb4d2aae66476fb"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.008004 24468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:11.022688 24468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:11.023480 24468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:11.023695 24468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:11.024442 24468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:11.025458 24468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:11.025532 24468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.025607 24468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:11.025651 24468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.035239 24468 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:34201
I20260810 23:07:11.035257 24598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:34201 every 8 connection(s)
I20260810 23:07:11.036254 24468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:11.041713 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24468
I20260810 23:07:11.041821 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:11.042119 14636 external_mini_cluster.cc:1428] Setting key fcd4e9be7619b4a74fcf30d56b048fd8
I20260810 23:07:11.044445 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--builtin_ntp_servers=127.14.75.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:07:11.046833 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.038400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:52059 (local address 127.14.75.62:42111)
0810 23:07:11.038557 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.038561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.039701 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 23:07:11.041257 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.041260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.041353 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.041470 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.042550 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.043227 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.044572 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.044866 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.044987 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.046302 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:11.046305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:11.046307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:11.046320 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:11.046341 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:11.046349 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:11.046432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.046600 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.046682 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:07:11.047785 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.038809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42111 (local address 127.14.75.2:52059)
0810 23:07:11.039523 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.039568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.040998 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.041667 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.041675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.041765 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:11.042395 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.042410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.043380 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.043383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.044433 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.044445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.045040 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.045045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.045226 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.045976 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:11.045977 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:11.046024 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:11.046100 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:11.046433 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:11.046446 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:11.046455 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.046600 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.046933 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260810 23:07:11.049189 24599 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42111
I20260810 23:07:11.049513 24599 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:11.050098 24599 heartbeater.cc:511] Master 127.14.75.62:42111 requested a full tablet report, sending...
I20260810 23:07:11.051409 24269 ts_manager.cc:194] Registered new tserver with Master: 62fd5fb0be2147b78e45f5707fff55c6 (127.14.75.2:34201)
I20260810 23:07:11.052349 24269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.2:52059
W20260810 23:07:11.180922 24602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:11.181241 24602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:11.181285 24602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:11.185536 24602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:11.185611 24602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:11.185711 24602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:07:11.190891 24602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:11.192072 24602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:11.193547 24602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:11.203984 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.204059 24602 server_base.cc:1034] running on GCE node
W20260810 23:07:11.203989 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:11.203995 24611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.204782 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:11.205569 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:11.206745 24602 hybrid_clock.cc:648] HybridClock initialized: now 1786403231206731 us; error 34 us; skew 500 ppm
I20260810 23:07:11.208747 24602 webserver.cc:460] Webserver started at http://127.14.75.3:33381/ using document root <none> and password file <none>
I20260810 23:07:11.209367 24602 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:11.209514 24602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:11.209774 24602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:11.212180 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "733541643ce145e7a557534b2a2baabd"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "c2925ec135e1c0d637addbf7afb2488d"
server_key_iv: "e35fe6be2fa934c7a3358d1438697495"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.212903 24602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "733541643ce145e7a557534b2a2baabd"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "c2925ec135e1c0d637addbf7afb2488d"
server_key_iv: "e35fe6be2fa934c7a3358d1438697495"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.216802 24602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 23:07:11.219599 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.220944 24602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:11.221082 24602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "733541643ce145e7a557534b2a2baabd"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "c2925ec135e1c0d637addbf7afb2488d"
server_key_iv: "e35fe6be2fa934c7a3358d1438697495"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.221216 24602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:11.234265 24602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:11.234970 24602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:11.235136 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:11.235710 24602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:11.236660 24602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:11.236717 24602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.236759 24602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:11.236781 24602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.246438 24602 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:37041
I20260810 23:07:11.246462 24732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:37041 every 8 connection(s)
I20260810 23:07:11.247428 24602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:11.252208 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24602
I20260810 23:07:11.252324 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:11.252578 14636 external_mini_cluster.cc:1428] Setting key e8b874eb1fcbeafc1d87f1dd859862a7
I20260810 23:07:11.257009 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.249289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53889 (local address 127.14.75.62:42111)
0810 23:07:11.249498 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.249503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.250588 (+ 1085us) server_negotiation.cc:407] Connection header received
0810 23:07:11.252031 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.252035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.252099 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.252181 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.253223 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.253934 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.255195 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.255410 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.255494 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.256533 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:11.256536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:11.256538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:11.256556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:11.256576 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:11.256584 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:11.256650 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.256798 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.256884 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:07:11.257928 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.249749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42111 (local address 127.14.75.3:53889)
0810 23:07:11.250428 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.250474 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.251813 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.252380 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.252389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.252494 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:11.253059 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.253078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.254075 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.254078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.255087 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.255096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.255553 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.255555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.255702 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.256278 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:11.256280 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:11.256321 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:11.256389 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:11.256670 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:11.256677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:11.256681 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.256780 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.257016 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260810 23:07:11.259150 24733 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42111
I20260810 23:07:11.259416 24733 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:11.259984 24733 heartbeater.cc:511] Master 127.14.75.62:42111 requested a full tablet report, sending...
I20260810 23:07:11.261087 24269 ts_manager.cc:194] Registered new tserver with Master: 733541643ce145e7a557534b2a2baabd (127.14.75.3:37041)
I20260810 23:07:11.261673 24269 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.3:53889
I20260810 23:07:11.266641 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:07:11.269706 24310 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:11.268634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.14.75.62:42111)
0810 23:07:11.268770 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.268774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.268992 (+ 218us) server_negotiation.cc:407] Connection header received
0810 23:07:11.269095 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.269097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.269153 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.269224 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.269546 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51102: BlockingRecv error: recv got EOF from 127.0.0.1:51102 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:07:11.269861 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24332
I20260810 23:07:11.277115 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24468
I20260810 23:07:11.283660 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24602
I20260810 23:07:11.290115 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24238
2026-08-10T23:07:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (954 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:11Z Disabled control of system clock
I20260810 23:07:11.321861 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:35547
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:07:11.436578 24744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:11.436888 24744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:11.436926 24744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:11.441188 24744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:11.441283 24744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:11.441299 24744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:11.441316 24744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:11.441334 24744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:11.446552 24744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:35547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:11.447780 24744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:11.449267 24744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:11.458171 24752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:11.458247 24749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:11.458187 24750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.458601 24744 server_base.cc:1034] running on GCE node
I20260810 23:07:11.459278 24744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:11.460479 24744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:11.461689 24744 hybrid_clock.cc:648] HybridClock initialized: now 1786403231461655 us; error 50 us; skew 500 ppm
I20260810 23:07:11.463858 24744 webserver.cc:460] Webserver started at http://127.14.75.62:39331/ using document root <none> and password file <none>
I20260810 23:07:11.464511 24744 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:11.464615 24744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:11.464865 24744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:11.467528 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "16e5d22ebd654441a5ec4a75c83785e0"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "68d836e5abbd13708b92bcc800642e34"
server_key_iv: "1dda8e857ac390f22c674597f5d3f8f2"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.468171 24744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "16e5d22ebd654441a5ec4a75c83785e0"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "68d836e5abbd13708b92bcc800642e34"
server_key_iv: "1dda8e857ac390f22c674597f5d3f8f2"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.472203 24744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:07:11.474952 24758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.476290 24744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:07:11.476440 24744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "16e5d22ebd654441a5ec4a75c83785e0"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "68d836e5abbd13708b92bcc800642e34"
server_key_iv: "1dda8e857ac390f22c674597f5d3f8f2"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.476588 24744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:11.481559 24744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:11.482234 24744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:11.482445 24744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:11.490299 24744 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:35547
I20260810 23:07:11.490310 24811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:35547 every 8 connection(s)
I20260810 23:07:11.491335 24744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:11.494794 24812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:11.498131 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24744
I20260810 23:07:11.498235 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:07:11.498521 14636 external_mini_cluster.cc:1428] Setting key 42f21ccf8197395aa1b896e22a4e041e
I20260810 23:07:11.501391 24812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0: Bootstrap starting.
I20260810 23:07:11.504262 24812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:11.504681 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.500085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.14.75.62:35547)
0810 23:07:11.500755 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.500772 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.500810 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:07:11.501518 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.501538 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.501957 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.502273 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.503218 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:11.503223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:11.503235 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:11.503245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:11.503311 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:11.503327 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:11.503445 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.503567 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.503866 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":159,"threads_started":1}
I20260810 23:07:11.505220 24812 log.cc:824] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:11.507573 24812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0: No bootstrap required, opened a new log
I20260810 23:07:11.511550 24812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } }
I20260810 23:07:11.511835 24812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:11.511870 24812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e5d22ebd654441a5ec4a75c83785e0, State: Initialized, Role: FOLLOWER
I20260810 23:07:11.512336 24812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } }
I20260810 23:07:11.512473 24812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:11.512517 24812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:11.512605 24812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:11.513691 24812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } }
I20260810 23:07:11.514019 24812 leader_election.cc:302] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e5d22ebd654441a5ec4a75c83785e0; no voters:
I20260810 23:07:11.514313 24812 leader_election.cc:288] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:11.514478 24817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:11.514755 24817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [term 1 LEADER]: Becoming Leader. State: Replica: 16e5d22ebd654441a5ec4a75c83785e0, State: Running, Role: LEADER
I20260810 23:07:11.515115 24817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } }
I20260810 23:07:11.515656 24812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:11.516901 24818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e5d22ebd654441a5ec4a75c83785e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } } }
I20260810 23:07:11.517050 24818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:11.517066 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e5d22ebd654441a5ec4a75c83785e0. Latest consensus state: current_term: 1 leader_uuid: "16e5d22ebd654441a5ec4a75c83785e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5d22ebd654441a5ec4a75c83785e0" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 35547 } } }
I20260810 23:07:11.517360 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:11.517515 24826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:11.520457 24826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:11.527035 24826 catalog_manager.cc:1347] Generated new cluster ID: d4c24b02cdb54ebfa9547f6ea297e180
I20260810 23:07:11.527125 24826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:11.543972 24826 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:11.544366 24826 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:11.550520 24826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e5d22ebd654441a5ec4a75c83785e0: Generated new TSK 0
I20260810 23:07:11.551271 24826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:11.559345 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--builtin_ntp_servers=127.14.75.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:07:11.680737 24836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:11.681102 24836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:11.681146 24836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:11.685490 24836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:11.685596 24836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:11.685673 24836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:07:11.690747 24836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:11.691859 24836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:11.693195 24836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:11.703732 24842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.703982 24836 server_base.cc:1034] running on GCE node
W20260810 23:07:11.703732 24841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:11.703732 24844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.704609 24836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:11.705276 24836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:11.706446 24836 hybrid_clock.cc:648] HybridClock initialized: now 1786403231706425 us; error 35 us; skew 500 ppm
I20260810 23:07:11.708392 24836 webserver.cc:460] Webserver started at http://127.14.75.1:41465/ using document root <none> and password file <none>
I20260810 23:07:11.708997 24836 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:11.709062 24836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:11.709290 24836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:11.711522 24836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "38e06be41a374cf2a0f43eafc548dacb"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "daad58cdf377e369074d009e801e1168"
server_key_iv: "7725d8813a7a46a0e9008cc6e662c013"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.712122 24836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "38e06be41a374cf2a0f43eafc548dacb"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "daad58cdf377e369074d009e801e1168"
server_key_iv: "7725d8813a7a46a0e9008cc6e662c013"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.716043 24836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:07:11.719007 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.720360 24836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:07:11.720500 24836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "38e06be41a374cf2a0f43eafc548dacb"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "daad58cdf377e369074d009e801e1168"
server_key_iv: "7725d8813a7a46a0e9008cc6e662c013"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.720623 24836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:11.725979 24836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:11.726651 24836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:11.726853 24836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:11.727445 24836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:11.728415 24836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:11.728463 24836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.728531 24836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:11.728562 24836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.738235 24836 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:36063
I20260810 23:07:11.738257 24965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:36063 every 8 connection(s)
I20260810 23:07:11.739241 24836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:11.745738 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.741094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:54987 (local address 127.14.75.62:35547)
0810 23:07:11.741245 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.741249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.742432 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 23:07:11.743819 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.743823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.743887 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.744001 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.745169 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:11.745172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:11.745175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:11.745189 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:11.745219 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:11.745228 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:11.745335 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.745522 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.745587 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:07:11.746775 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.741576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:35547 (local address 127.14.75.1:54987)
0810 23:07:11.742258 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.742302 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.743586 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.744204 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.744211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.744304 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:11.744895 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:11.744896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:11.744937 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:11.745010 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:11.745314 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:11.745323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:11.745329 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.745565 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.745818 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260810 23:07:11.746829 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24836
I20260810 23:07:11.746925 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:11.747242 14636 external_mini_cluster.cc:1428] Setting key f08772e7d95dc9432d672ab4aa343b42
I20260810 23:07:11.748173 24966 heartbeater.cc:347] Connected to a master server at 127.14.75.62:35547
I20260810 23:07:11.748413 24966 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:11.748868 24966 heartbeater.cc:511] Master 127.14.75.62:35547 requested a full tablet report, sending...
I20260810 23:07:11.749378 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--builtin_ntp_servers=127.14.75.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:07:11.750450 24775 ts_manager.cc:194] Registered new tserver with Master: 38e06be41a374cf2a0f43eafc548dacb (127.14.75.1:36063)
W20260810 23:07:11.867198 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:11.867493 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:11.867535 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:11.871680 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:11.871748 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:11.871830 24969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:07:11.876776 24969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:11.877954 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:11.879324 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:11.889639 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:11.889634 24977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.889703 24969 server_base.cc:1034] running on GCE node
W20260810 23:07:11.889631 24974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:11.890357 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:11.891052 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:11.892221 24969 hybrid_clock.cc:648] HybridClock initialized: now 1786403231892209 us; error 28 us; skew 500 ppm
I20260810 23:07:11.894284 24969 webserver.cc:460] Webserver started at http://127.14.75.2:39089/ using document root <none> and password file <none>
I20260810 23:07:11.894898 24969 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:11.894968 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:11.895228 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:11.897459 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "a0a9fad8b59b4b48a0232f097517f211"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "a6c9a6d63d6e9b2fc20bf186801e1d6e"
server_key_iv: "1533cc2816f0bca3482c90db6e058998"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.898147 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "a0a9fad8b59b4b48a0232f097517f211"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "a6c9a6d63d6e9b2fc20bf186801e1d6e"
server_key_iv: "1533cc2816f0bca3482c90db6e058998"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.902208 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:07:11.904872 24983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.906159 24969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:07:11.906287 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "a0a9fad8b59b4b48a0232f097517f211"
format_stamp: "Formatted at 2026-08-10 23:07:11 on dist-test-slave-srgv"
server_key: "a6c9a6d63d6e9b2fc20bf186801e1d6e"
server_key_iv: "1533cc2816f0bca3482c90db6e058998"
server_key_version: "encryptionkey@0"
I20260810 23:07:11.906405 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:11.911127 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:11.911770 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:11.911958 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:11.912544 24969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:11.913548 24969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:11.913604 24969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.913678 24969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:11.913719 24969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:11.922866 24969 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37515
I20260810 23:07:11.922890 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37515 every 8 connection(s)
I20260810 23:07:11.923907 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:11.926610 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 24969
I20260810 23:07:11.926754 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:11.927093 14636 external_mini_cluster.cc:1428] Setting key 8ce38cfc1744b105e821dbacaa343744
I20260810 23:07:11.929289 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--builtin_ntp_servers=127.14.75.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:07:11.930939 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.925985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:56379 (local address 127.14.75.62:35547)
0810 23:07:11.926169 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.926175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.927527 (+ 1352us) server_negotiation.cc:407] Connection header received
0810 23:07:11.928991 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.928996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.929045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.929136 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:11.930384 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:11.930390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:11.930396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:11.930414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:11.930444 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:11.930458 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:11.930571 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.930696 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.930751 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:07:11.932121 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.926527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:35547 (local address 127.14.75.2:56379)
0810 23:07:11.927336 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.927380 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.928770 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.929297 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.929303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.929456 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:11.930105 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:11.930107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:11.930150 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:11.930222 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:11.930551 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:11.930557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:11.930561 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.930724 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.931019 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":157,"threads_started":1}
I20260810 23:07:11.933352 25100 heartbeater.cc:347] Connected to a master server at 127.14.75.62:35547
I20260810 23:07:11.933627 25100 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:11.934068 25100 heartbeater.cc:511] Master 127.14.75.62:35547 requested a full tablet report, sending...
I20260810 23:07:11.935271 24775 ts_manager.cc:194] Registered new tserver with Master: a0a9fad8b59b4b48a0232f097517f211 (127.14.75.2:37515)
W20260810 23:07:12.050191 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:12.050601 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:12.050697 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:12.055249 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:12.055315 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:12.055419 25103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:07:12.060583 25103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:12.061856 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:12.063298 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:12.073170 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.073172 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.073311 25103 server_base.cc:1034] running on GCE node
W20260810 23:07:12.073172 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.073902 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:12.074621 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:12.075773 25103 hybrid_clock.cc:648] HybridClock initialized: now 1786403232075762 us; error 31 us; skew 500 ppm
I20260810 23:07:12.077826 25103 webserver.cc:460] Webserver started at http://127.14.75.3:33173/ using document root <none> and password file <none>
I20260810 23:07:12.078456 25103 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:12.078526 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:12.078749 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:12.081054 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "e155ee0dd20e4098bbaa8dfa40597f94"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-srgv"
server_key: "dafcf144c558f12544c0d6c565bd8aa7"
server_key_iv: "4c46d65355883998ef4c68f3cf12528f"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.081765 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "e155ee0dd20e4098bbaa8dfa40597f94"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-srgv"
server_key: "dafcf144c558f12544c0d6c565bd8aa7"
server_key_iv: "4c46d65355883998ef4c68f3cf12528f"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.085686 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:07:12.088541 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.090301 25103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:12.090441 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "e155ee0dd20e4098bbaa8dfa40597f94"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-srgv"
server_key: "dafcf144c558f12544c0d6c565bd8aa7"
server_key_iv: "4c46d65355883998ef4c68f3cf12528f"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.090588 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:12.095620 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:12.096267 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:12.096458 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:12.097029 25103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:12.098052 25103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:12.098100 25103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.098140 25103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:12.098161 25103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.106904 25103 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:34563
I20260810 23:07:12.106952 25232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:34563 every 8 connection(s)
I20260810 23:07:12.107859 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:12.114001 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.109746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:46855 (local address 127.14.75.62:35547)
0810 23:07:12.109892 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:12.109896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:12.111026 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 23:07:12.112314 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:12.112317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:12.112358 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:12.112437 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:12.113592 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:12.113595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:12.113597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:12.113609 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:12.113638 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:12.113647 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:12.113729 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:12.113849 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:07:12.113889 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:07:12.114856 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.110151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:35547 (local address 127.14.75.3:46855)
0810 23:07:12.110861 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:12.110902 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:12.112113 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:12.112599 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:12.112605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:12.112695 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:12.113276 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:12.113278 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:12.113316 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:12.113382 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:12.113727 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:12.113734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:12.113738 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:12.113872 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:07:12.114063 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260810 23:07:12.116003 25233 heartbeater.cc:347] Connected to a master server at 127.14.75.62:35547
I20260810 23:07:12.116325 25233 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:12.116766 25233 heartbeater.cc:511] Master 127.14.75.62:35547 requested a full tablet report, sending...
I20260810 23:07:12.117161 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 25103
I20260810 23:07:12.117290 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:12.117650 14636 external_mini_cluster.cc:1428] Setting key f0d6db6eef72db0f6eeafcef4f97a08d
I20260810 23:07:12.118054 24775 ts_manager.cc:194] Registered new tserver with Master: e155ee0dd20e4098bbaa8dfa40597f94 (127.14.75.3:34563)
I20260810 23:07:12.119827 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:07:12.122726 24815 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:12.121911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.14.75.62:35547)
0810 23:07:12.122067 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:12.122072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:12.122190 (+ 118us) server_negotiation.cc:407] Connection header received
0810 23:07:12.122283 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:12.122285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:12.122316 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:12.122381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:12.122592 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37286: BlockingRecv error: recv got EOF from 127.0.0.1:37286 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:07:12.122846 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24836
I20260810 23:07:12.130539 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24969
I20260810 23:07:12.138731 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 25103
I20260810 23:07:12.145864 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 24744
2026-08-10T23:07:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (855 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:12 dist-test-slave-srgv krb5kdc[25243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:12 dist-test-slave-srgv krb5kdc[25243](info): set up 2 sockets
Aug 10 23:07:12 dist-test-slave-srgv krb5kdc[25243](info): commencing operation
krb5kdc: starting...
W20260810 23:07:13.203290 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:07:13.332726 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40713
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:13.445931 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.446267 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.446341 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.450492 25257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:13.450605 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.450644 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.450711 25257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:13.450747 25257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:13.456352 25257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:40713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:13.457558 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.458626 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.467662 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.467641 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.467638 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.467715 25257 server_base.cc:1034] running on GCE node
I20260810 23:07:13.468557 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.469746 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.470909 25257 hybrid_clock.cc:648] HybridClock initialized: now 1786403233470881 us; error 42 us; skew 500 ppm
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:13.473721 25257 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:07:13.474697 25257 webserver.cc:460] Webserver started at http://127.14.75.62:37739/ using document root <none> and password file <none>
I20260810 23:07:13.475265 25257 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.475361 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.475600 25257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.477788 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "8f9c4d0faf04444a99dd577119a01dda"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "1b4c7d3b6c4b2d4694d390705f02c739"
server_key_iv: "4c4c0c30f320f09fb9f89b38cb11b844"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.478389 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "8f9c4d0faf04444a99dd577119a01dda"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "1b4c7d3b6c4b2d4694d390705f02c739"
server_key_iv: "4c4c0c30f320f09fb9f89b38cb11b844"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.482297 25257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:07:13.484802 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.486115 25257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:07:13.486289 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "8f9c4d0faf04444a99dd577119a01dda"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "1b4c7d3b6c4b2d4694d390705f02c739"
server_key_iv: "4c4c0c30f320f09fb9f89b38cb11b844"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.486418 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.508260 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.511606 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.511839 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.519316 25257 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:40713
I20260810 23:07:13.519629 25325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:40713 every 8 connection(s)
I20260810 23:07:13.520385 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:13.523403 25326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.529055 25326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: Bootstrap starting.
I20260810 23:07:13.529975 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 25257
I20260810 23:07:13.530103 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:07:13.530405 14636 external_mini_cluster.cc:1428] Setting key 316657114661076cbef9ba5a7528ed13
I20260810 23:07:13.532253 25326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.533015 25326 log.cc:824] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.535511 25326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: No bootstrap required, opened a new log
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:07:13.540383 25326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } }
I20260810 23:07:13.540627 25326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.540704 25326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9c4d0faf04444a99dd577119a01dda, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.541157 25326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } }
I20260810 23:07:13.541280 25326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:13.541365 25326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:13.541560 25326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.542587 25326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } }
I20260810 23:07:13.542948 25326 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9c4d0faf04444a99dd577119a01dda; no voters:
I20260810 23:07:13.543335 25326 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:13.543412 25331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:13.543627 25331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [term 1 LEADER]: Becoming Leader. State: Replica: 8f9c4d0faf04444a99dd577119a01dda, State: Running, Role: LEADER
I20260810 23:07:13.544039 25331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } }
I20260810 23:07:13.544795 25326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:13.545909 25332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9c4d0faf04444a99dd577119a01dda. Latest consensus state: current_term: 1 leader_uuid: "8f9c4d0faf04444a99dd577119a01dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } } }
I20260810 23:07:13.546064 25332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:13.546344 25333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9c4d0faf04444a99dd577119a01dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9c4d0faf04444a99dd577119a01dda" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40713 } } }
I20260810 23:07:13.546480 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.531717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.14.75.62:40713)
0810 23:07:13.532144 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.532160 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.532203 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:07:13.532924 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.532942 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.533345 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.533775 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.534688 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.535923 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.537079 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.537654 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.537780 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.540837 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.540842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.540857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:13.540886 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.542962 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.543468 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.543477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.543485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.543667 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.544004 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.544009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.544012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.544511 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.544729 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.545048 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.545393 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":154,"threads_started":1}
I20260810 23:07:13.546433 25340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:13.546521 25333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:13.550496 25340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:13.556962 25340 catalog_manager.cc:1347] Generated new cluster ID: 7b19f0628b8b4b64bbce26c90f332db6
I20260810 23:07:13.557056 25340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:13.564483 25340 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:13.565888 25340 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:13.584241 25340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: Generated new TSK 0
I20260810 23:07:13.585033 25340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:07:13.616822 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:13.733744 25352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.734073 25352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.734162 25352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.738241 25352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.738348 25352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.738444 25352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:07:13.743801 25352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:13.744956 25352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.746199 25352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.756660 25358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.756726 25352 server_base.cc:1034] running on GCE node
W20260810 23:07:13.756654 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.756649 25357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.757329 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.758030 25352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.759220 25352 hybrid_clock.cc:648] HybridClock initialized: now 1786403233759187 us; error 47 us; skew 500 ppm
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:13.762074 25352 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:07:13.762972 25352 webserver.cc:460] Webserver started at http://127.14.75.1:38949/ using document root <none> and password file <none>
I20260810 23:07:13.763710 25352 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.763806 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.764050 25352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.766173 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "f3d1bb7dc6de4e7cb10934e526d53a52"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "fe807851f0d8c70bfb17147af6e65a4e"
server_key_iv: "babd9f57d75ebc3348373b40beda8f20"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.766712 25352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "f3d1bb7dc6de4e7cb10934e526d53a52"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "fe807851f0d8c70bfb17147af6e65a4e"
server_key_iv: "babd9f57d75ebc3348373b40beda8f20"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.770612 25352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 23:07:13.773559 25367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.774829 25352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:07:13.774961 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "f3d1bb7dc6de4e7cb10934e526d53a52"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "fe807851f0d8c70bfb17147af6e65a4e"
server_key_iv: "babd9f57d75ebc3348373b40beda8f20"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.775087 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.797703 25352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.800933 25352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.801162 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.801914 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:13.802829 25352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:13.802879 25352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.802953 25352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:13.802983 25352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.812294 25352 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:39357
I20260810 23:07:13.812318 25481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:39357 every 8 connection(s)
I20260810 23:07:13.813237 25352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:13.814328 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 25352
I20260810 23:07:13.814450 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:13.814783 14636 external_mini_cluster.cc:1428] Setting key d4aa527bdaf2ed21d13d3e50dccc7064
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:07:13.829869 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.815257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:48413 (local address 127.14.75.62:40713)
0810 23:07:13.815408 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.815412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.816498 (+ 1086us) server_negotiation.cc:407] Connection header received
0810 23:07:13.818669 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.818675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.818746 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.818859 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.820115 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.821022 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.822387 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.822857 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.823003 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.826054 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.826057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.826059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:13.826088 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.827513 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.828120 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.828123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.828124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.828206 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.828617 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.828621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.828625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.828841 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.828956 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.829543 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.829681 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:07:13.831051 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.815731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40713 (local address 127.14.75.1:48413)
0810 23:07:13.816309 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.816388 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.818371 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.819039 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.819046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.819433 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:13.819940 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.819954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.821168 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.821171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.822242 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.822252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.823030 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.823033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.823184 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.823907 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:13.823925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:13.825838 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:13.827647 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:13.827675 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:13.827698 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:13.828008 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:13.828305 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:13.828311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:13.828317 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:13.828505 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:13.828956 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:13.828965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:13.829255 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.829489 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.829846 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":157,"threads_started":1}
I20260810 23:07:13.832598 25482 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40713
I20260810 23:07:13.832865 25482 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:13.833395 25482 heartbeater.cc:511] Master 127.14.75.62:40713 requested a full tablet report, sending...
I20260810 23:07:13.835098 25289 ts_manager.cc:194] Registered new tserver with Master: f3d1bb7dc6de4e7cb10934e526d53a52 (127.14.75.1:39357)
I20260810 23:07:13.836265 25289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:48413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:07:13.846014 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:13.964483 25487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.964762 25487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.964807 25487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.968888 25487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.968959 25487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.969022 25487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:07:13.974481 25487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:13.975523 25487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.976502 25487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.986389 25493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.986414 25492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.986414 25495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.986809 25487 server_base.cc:1034] running on GCE node
I20260810 23:07:13.987251 25487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.987932 25487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.989096 25487 hybrid_clock.cc:648] HybridClock initialized: now 1786403233989068 us; error 41 us; skew 500 ppm
Aug 10 23:07:13 dist-test-slave-srgv krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:13.992028 25487 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:07:13.993019 25487 webserver.cc:460] Webserver started at http://127.14.75.2:34993/ using document root <none> and password file <none>
I20260810 23:07:13.993678 25487 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.993770 25487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.994004 25487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.996093 25487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "55793ca3968c48e65676a5d59e341c37"
server_key_iv: "ac672f56dc5f221e2664a9e7735de8ce"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.996771 25487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "55793ca3968c48e65676a5d59e341c37"
server_key_iv: "ac672f56dc5f221e2664a9e7735de8ce"
server_key_version: "encryptionkey@0"
I20260810 23:07:14.000665 25487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:07:14.003331 25503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.004613 25487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:07:14.004760 25487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-srgv"
server_key: "55793ca3968c48e65676a5d59e341c37"
server_key_iv: "ac672f56dc5f221e2664a9e7735de8ce"
server_key_version: "encryptionkey@0"
I20260810 23:07:14.004879 25487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:14.020372 25487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:14.023478 25487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:14.023717 25487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:14.024384 25487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:14.025260 25487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:14.025336 25487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.025436 25487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:14.025493 25487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.035297 25487 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37899
I20260810 23:07:14.035318 25619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37899 every 8 connection(s)
I20260810 23:07:14.036275 25487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:14.043841 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 25487
I20260810 23:07:14.043991 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:14.044302 14636 external_mini_cluster.cc:1428] Setting key 7f531689bca662cc7c5c8fffb41e361d
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403233, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:07:14.051849 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.038220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:48969 (local address 127.14.75.62:40713)
0810 23:07:14.038364 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.038368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.039235 (+ 867us) server_negotiation.cc:407] Connection header received
0810 23:07:14.040677 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.040680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.040741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.040822 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:14.042113 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.042663 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.043991 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.044279 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.044409 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.047431 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:14.047434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:14.047437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:14.047472 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:14.049314 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.050046 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.050049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.050050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.050128 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.050548 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.050550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.050552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.050749 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:14.050828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.051603 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.051721 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:07:14.053320 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.038600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40713 (local address 127.14.75.2:48969)
0810 23:07:14.039068 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.039113 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.040435 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.041006 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.041013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.041390 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:14.041962 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.041975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.042806 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.042809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.043840 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.043852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.044466 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.044470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.044672 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.045399 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:14.045445 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:14.047252 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:14.049484 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:14.049494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:14.049516 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:14.049939 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:14.050225 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:14.050231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:14.050236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:14.050445 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:14.050874 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:14.050884 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:14.051295 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.051539 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.051971 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260810 23:07:14.054656 25620 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40713
I20260810 23:07:14.054937 25620 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:14.055473 25620 heartbeater.cc:511] Master 127.14.75.62:40713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:07:14.056633 25289 ts_manager.cc:194] Registered new tserver with Master: 1dfd0cf1cdca4b88a192d6fbaec5eed2 (127.14.75.2:37899)
I20260810 23:07:14.057195 25289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:48969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:07:14.075165 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:14.194463 25625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:14.194734 25625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:14.194808 25625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:14.198935 25625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:14.199013 25625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:14.199074 25625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:07:14.204674 25625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:14.205948 25625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:14.207023 25625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:14.216857 25633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:14.216857 25630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:14.216958 25631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:14.217183 25625 server_base.cc:1034] running on GCE node
I20260810 23:07:14.217674 25625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:14.218372 25625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:14.219565 25625 hybrid_clock.cc:648] HybridClock initialized: now 1786403234219527 us; error 52 us; skew 500 ppm
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:14.222462 25625 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:07:14.223384 25625 webserver.cc:460] Webserver started at http://127.14.75.3:40543/ using document root <none> and password file <none>
I20260810 23:07:14.223913 25625 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:14.223974 25625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:14.224167 25625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:14.226305 25625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "1a7c59aec6614b1fab4eceb4a7b60822"
format_stamp: "Formatted at 2026-08-10 23:07:14 on dist-test-slave-srgv"
server_key: "46a0ba7df594391f8995d44fd2dad8fe"
server_key_iv: "6e626fa59f6e8845247a9c2802bad744"
server_key_version: "encryptionkey@0"
I20260810 23:07:14.226819 25625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "1a7c59aec6614b1fab4eceb4a7b60822"
format_stamp: "Formatted at 2026-08-10 23:07:14 on dist-test-slave-srgv"
server_key: "46a0ba7df594391f8995d44fd2dad8fe"
server_key_iv: "6e626fa59f6e8845247a9c2802bad744"
server_key_version: "encryptionkey@0"
I20260810 23:07:14.230633 25625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:07:14.233183 25641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.234532 25625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:07:14.234673 25625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "1a7c59aec6614b1fab4eceb4a7b60822"
format_stamp: "Formatted at 2026-08-10 23:07:14 on dist-test-slave-srgv"
server_key: "46a0ba7df594391f8995d44fd2dad8fe"
server_key_iv: "6e626fa59f6e8845247a9c2802bad744"
server_key_version: "encryptionkey@0"
I20260810 23:07:14.234792 25625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:14.246142 25625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:14.249325 25625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:14.249590 25625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:14.250217 25625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:14.251067 25625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:14.251116 25625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.251210 25625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:14.251248 25625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:14.260172 25625 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:40309
I20260810 23:07:14.260193 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:40309 every 8 connection(s)
I20260810 23:07:14.261127 25625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:14.262548 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 25625
I20260810 23:07:14.262688 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:14.263031 14636 external_mini_cluster.cc:1428] Setting key 6c8a9057dfbe1335a3bffe65f8f0f2d4
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403234, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:07:14.275200 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.263055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:35739 (local address 127.14.75.62:40713)
0810 23:07:14.263233 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.263237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.264355 (+ 1118us) server_negotiation.cc:407] Connection header received
0810 23:07:14.266078 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.266081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.266134 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.266203 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:14.267330 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.267853 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.269020 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.269231 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.269307 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.271982 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:14.271985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:14.271987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:14.272010 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:14.273329 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.273857 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.273859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.273860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.273935 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.274237 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.274239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.274240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.274417 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:14.274491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.274963 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.275064 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:07:14.276101 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.263609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40713 (local address 127.14.75.3:35739)
0810 23:07:14.264204 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.264250 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.265869 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.266345 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.266352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.266716 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:14.267204 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.267217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.267963 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.267965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.268916 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.268925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.269361 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.269364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.269624 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.270331 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:14.270348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:14.271816 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:14.273487 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:14.273492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:14.273505 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:14.273768 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:14.274016 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:14.274018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:14.274019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:14.274150 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:14.274501 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:14.274506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:14.274750 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.274901 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.275158 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260810 23:07:14.277561 25759 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40713
I20260810 23:07:14.277776 25759 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:14.278241 25759 heartbeater.cc:511] Master 127.14.75.62:40713 requested a full tablet report, sending...
I20260810 23:07:14.279243 25289 ts_manager.cc:194] Registered new tserver with Master: 1a7c59aec6614b1fab4eceb4a7b60822 (127.14.75.3:40309)
I20260810 23:07:14.279831 25289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:35739
I20260810 23:07:14.289512 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:14.298611 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.291494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.14.75.62:40713)
0810 23:07:14.291655 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.291660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.291774 (+ 114us) server_negotiation.cc:407] Connection header received
0810 23:07:14.291873 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.291875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.291917 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.291985 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:14.292639 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.293148 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.294002 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.294217 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.294295 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.295681 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:14.295684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:14.295686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:14.295703 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:14.296947 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.297331 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.297334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.297335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.297527 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:14.297776 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:14.297779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:14.297780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:14.297999 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:14.298103 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.298304 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.298404 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:07:14.301707 25289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:14.304904 25289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:14.319648 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.314507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.14.75.2:37899)
0810 23:07:14.315027 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.315038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.315060 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:07:14.315133 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.315138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.315322 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.315470 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:14.315674 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.316722 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.318364 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.318908 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319041 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.319200 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.319380 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.319478 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":137,"threads_started":1}
I20260810 23:07:14.319664 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.313895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37899 (local address 127.0.0.1:46696)
0810 23:07:14.314522 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.314536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.314618 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.315447 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.315450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.315468 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:14.315571 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.315576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.316848 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.316853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.318199 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.318210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.319041 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.319163 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.319203 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.319337 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.319466 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":11,"thread_start_us":101,"threads_started":1}
I20260810 23:07:14.319698 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.314313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40309 (local address 127.0.0.1:51088)
0810 23:07:14.314995 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.315012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.315100 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.315780 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.315782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.315799 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:14.315890 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.315897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.317005 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.317007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.318219 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.318225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.319235 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.319369 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.319391 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.319457 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.319528 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":58,"thread_start_us":77,"threads_started":1}
I20260810 23:07:14.319804 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.314850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.14.75.3:40309)
0810 23:07:14.315349 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.315364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.315387 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:07:14.315463 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.315468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.315646 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.315790 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:14.316018 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.316866 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.318333 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.319085 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319379 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.319521 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.319570 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.319652 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":172,"threads_started":1}
I20260810 23:07:14.320304 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.313608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39357 (local address 127.0.0.1:59614)
0810 23:07:14.313963 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.314020 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.314154 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.314929 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.314935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.314975 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:14.315112 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.315119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.316600 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.316603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.317908 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.317917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.319235 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.320030 (+ 791us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.320053 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.320121 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.320190 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":61,"thread_start_us":86,"threads_started":1}
I20260810 23:07:14.320824 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.313709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.14.75.1:39357)
0810 23:07:14.314106 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.314123 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.314147 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:14.314370 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.314377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.314578 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.314737 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:14.315260 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.316460 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.318048 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.318902 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.319011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.319172 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.320561 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.320667 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
I20260810 23:07:14.322695 25691 tablet_service.cc:1467] Processing CreateTablet for tablet a1a04e5bcefb45a181973e942650aa74 (DEFAULT_TABLE table=test-table [id=19d2ab0472b24376a3583498d7048894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:14.322932 25553 tablet_service.cc:1467] Processing CreateTablet for tablet a1a04e5bcefb45a181973e942650aa74 (DEFAULT_TABLE table=test-table [id=19d2ab0472b24376a3583498d7048894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:14.323747 25691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a04e5bcefb45a181973e942650aa74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:14.323390 25416 tablet_service.cc:1467] Processing CreateTablet for tablet a1a04e5bcefb45a181973e942650aa74 (DEFAULT_TABLE table=test-table [id=19d2ab0472b24376a3583498d7048894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:14.323953 25553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a04e5bcefb45a181973e942650aa74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:14.324335 25416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a04e5bcefb45a181973e942650aa74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:14.330734 25777 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Bootstrap starting.
I20260810 23:07:14.331933 25780 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: Bootstrap starting.
I20260810 23:07:14.333288 25777 tablet_bootstrap.cc:654] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:14.334179 25780 tablet_bootstrap.cc:654] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:14.334585 25778 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Bootstrap starting.
I20260810 23:07:14.334756 25777 log.cc:824] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:14.334985 25780 log.cc:824] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:14.336861 25780 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: No bootstrap required, opened a new log
I20260810 23:07:14.337050 25777 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: No bootstrap required, opened a new log
I20260810 23:07:14.337092 25780 ts_tablet_manager.cc:1397] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:07:14.337297 25777 ts_tablet_manager.cc:1397] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:07:14.337318 25778 tablet_bootstrap.cc:654] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:14.338260 25778 log.cc:824] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:14.340631 25778 tablet_bootstrap.cc:492] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: No bootstrap required, opened a new log
I20260810 23:07:14.340848 25778 ts_tablet_manager.cc:1397] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:07:14.341161 25780 raft_consensus.cc:348] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.341418 25780 raft_consensus.cc:374] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:14.341516 25780 raft_consensus.cc:729] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d1bb7dc6de4e7cb10934e526d53a52, State: Initialized, Role: FOLLOWER
I20260810 23:07:14.341351 25777 raft_consensus.cc:348] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.341625 25777 raft_consensus.cc:374] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:14.341714 25777 raft_consensus.cc:729] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7c59aec6614b1fab4eceb4a7b60822, State: Initialized, Role: FOLLOWER
I20260810 23:07:14.342079 25780 consensus_queue.cc:260] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.342134 25777 consensus_queue.cc:260] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.343112 25780 ts_tablet_manager.cc:1428] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:07:14.343125 25777 ts_tablet_manager.cc:1428] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:07:14.343487 25759 heartbeater.cc:503] Master 127.14.75.62:40713 was elected leader, sending a full tablet report...
I20260810 23:07:14.343526 25482 heartbeater.cc:503] Master 127.14.75.62:40713 was elected leader, sending a full tablet report...
I20260810 23:07:14.344849 25778 raft_consensus.cc:348] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.345118 25778 raft_consensus.cc:374] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:14.345230 25778 raft_consensus.cc:729] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfd0cf1cdca4b88a192d6fbaec5eed2, State: Initialized, Role: FOLLOWER
I20260810 23:07:14.345709 25778 consensus_queue.cc:260] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.346800 25778 ts_tablet_manager.cc:1428] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:07:14.346988 25620 heartbeater.cc:503] Master 127.14.75.62:40713 was elected leader, sending a full tablet report...
I20260810 23:07:14.409649 25784 raft_consensus.cc:482] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:14.409940 25784 raft_consensus.cc:504] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.411360 25784 leader_election.cc:288] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a7c59aec6614b1fab4eceb4a7b60822 (127.14.75.3:40309), 1dfd0cf1cdca4b88a192d6fbaec5eed2 (127.14.75.2:37899)
I20260810 23:07:14.415356 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.411744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:34799 (local address 127.14.75.2:37899)
0810 23:07:14.411926 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.411931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.412111 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:07:14.412281 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.412284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.412333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.412420 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:14.412692 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.413182 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414352 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.414756 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.414944 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.415129 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.415208 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:07:14.415373 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.411728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:49461 (local address 127.14.75.3:40309)
0810 23:07:14.411985 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.411993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.412026 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:07:14.412193 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.412196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.412252 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.412334 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:14.412623 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.413351 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414392 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.414797 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414915 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.414977 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.415166 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.415242 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 23:07:14.415400 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.411612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37899 (local address 127.14.75.1:34799)
0810 23:07:14.412012 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.412023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.412162 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.412431 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.412434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.412450 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:14.412575 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.412580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.413298 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.413302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.414218 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.414227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.414894 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.414991 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.415012 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.415112 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.415210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260810 23:07:14.415408 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.411582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40309 (local address 127.14.75.1:49461)
0810 23:07:14.411949 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.411964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:14.412063 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:14.412358 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:14.412361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:14.412377 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:14.412512 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.412518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.413503 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.413509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.414266 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:14.414275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.414918 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.414921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:14.415043 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.415060 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:07:14.415147 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:07:14.415248 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":22,"thread_start_us":153,"threads_started":1}
I20260810 23:07:14.416071 25573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a04e5bcefb45a181973e942650aa74" candidate_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" is_pre_election: true
I20260810 23:07:14.416384 25712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a04e5bcefb45a181973e942650aa74" candidate_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" is_pre_election: true
I20260810 23:07:14.416447 25573 raft_consensus.cc:2454] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d1bb7dc6de4e7cb10934e526d53a52 in term 0.
I20260810 23:07:14.416682 25712 raft_consensus.cc:2454] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d1bb7dc6de4e7cb10934e526d53a52 in term 0.
I20260810 23:07:14.416970 25369 leader_election.cc:302] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfd0cf1cdca4b88a192d6fbaec5eed2, f3d1bb7dc6de4e7cb10934e526d53a52; no voters:
I20260810 23:07:14.417282 25784 raft_consensus.cc:2781] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:14.417368 25784 raft_consensus.cc:482] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:14.417430 25784 raft_consensus.cc:3037] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:14.418511 25784 raft_consensus.cc:504] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.418849 25784 leader_election.cc:288] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [CANDIDATE]: Term 1 election: Requested vote from peers 1a7c59aec6614b1fab4eceb4a7b60822 (127.14.75.3:40309), 1dfd0cf1cdca4b88a192d6fbaec5eed2 (127.14.75.2:37899)
I20260810 23:07:14.419162 25712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a04e5bcefb45a181973e942650aa74" candidate_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7c59aec6614b1fab4eceb4a7b60822"
I20260810 23:07:14.419272 25573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a04e5bcefb45a181973e942650aa74" candidate_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2"
I20260810 23:07:14.419284 25712 raft_consensus.cc:3037] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:14.419414 25573 raft_consensus.cc:3037] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:14.420404 25712 raft_consensus.cc:2454] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d1bb7dc6de4e7cb10934e526d53a52 in term 1.
I20260810 23:07:14.420552 25573 raft_consensus.cc:2454] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d1bb7dc6de4e7cb10934e526d53a52 in term 1.
I20260810 23:07:14.420745 25368 leader_election.cc:302] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a7c59aec6614b1fab4eceb4a7b60822, f3d1bb7dc6de4e7cb10934e526d53a52; no voters:
I20260810 23:07:14.421033 25784 raft_consensus.cc:2781] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:14.421227 25784 raft_consensus.cc:686] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 1 LEADER]: Becoming Leader. State: Replica: f3d1bb7dc6de4e7cb10934e526d53a52, State: Running, Role: LEADER
I20260810 23:07:14.421603 25784 consensus_queue.cc:237] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } }
I20260810 23:07:14.424791 25289 catalog_manager.cc:5495] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3d1bb7dc6de4e7cb10934e526d53a52 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1bb7dc6de4e7cb10934e526d53a52" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 } health_report { overall_health: UNKNOWN } } }
I20260810 23:07:14.442665 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.439319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.14.75.1:39357)
0810 23:07:14.439447 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:14.439451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:14.439465 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:07:14.439590 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:14.439592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:14.439649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:14.439740 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:14.439970 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.440467 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.441280 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:14.441552 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:14.441636 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:14.441918 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:14.442347 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:14.442437 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:14.442500 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:07:14.442551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:07:14.448849 25573 raft_consensus.cc:1261] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 1 FOLLOWER]: Refusing update from remote peer f3d1bb7dc6de4e7cb10934e526d53a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:14.448850 25712 raft_consensus.cc:1261] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 1 FOLLOWER]: Refusing update from remote peer f3d1bb7dc6de4e7cb10934e526d53a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:14.449649 25790 consensus_queue.cc:1035] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dfd0cf1cdca4b88a192d6fbaec5eed2" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:14.449898 25784 consensus_queue.cc:1035] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a7c59aec6614b1fab4eceb4a7b60822" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:14.458947 25794 mvcc.cc:205] Tried to move back new op lower bound from 7317107648294604800 to 7317107648193589248. Current Snapshot: MvccSnapshot[applied={T|T < 7317107648294604800}]
I20260810 23:07:14.467499 25289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:14.467711 25289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:14.470265 25289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f9c4d0faf04444a99dd577119a01dda: Sending DeleteTablet for 3 replicas of tablet a1a04e5bcefb45a181973e942650aa74
I20260810 23:07:14.471205 25553 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a04e5bcefb45a181973e942650aa74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:14 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46696
I20260810 23:07:14.471401 25691 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a04e5bcefb45a181973e942650aa74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:14 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:51088
I20260810 23:07:14.471412 25416 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a04e5bcefb45a181973e942650aa74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:14 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:59614
I20260810 23:07:14.471808 25804 tablet_replica.cc:330] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: stopping tablet replica
I20260810 23:07:14.471952 25805 tablet_replica.cc:330] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: stopping tablet replica
I20260810 23:07:14.472210 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 25352
I20260810 23:07:14.472214 25806 tablet_replica.cc:330] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52: stopping tablet replica
I20260810 23:07:14.472232 25804 raft_consensus.cc:2229] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:14.472515 25806 raft_consensus.cc:2229] T a1a04e5bcefb45a181973e942650aa74 P f3d1bb7dc6de4e7cb10934e526d53a52 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:07:14.473453 25805 raft_consensus.cc:2229] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:14.473532 25804 raft_consensus.cc:2258] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:14.474181 25805 raft_consensus.cc:2258] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:14.476332 25804 ts_tablet_manager.cc:1905] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:14.476758 25805 ts_tablet_manager.cc:1905] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:14.479099 25805 ts_tablet_manager.cc:1918] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:14.479229 25805 log.cc:1196] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/a1a04e5bcefb45a181973e942650aa74
I20260810 23:07:14.479568 25805 ts_tablet_manager.cc:1939] T a1a04e5bcefb45a181973e942650aa74 P 1a7c59aec6614b1fab4eceb4a7b60822: Deleting consensus metadata
I20260810 23:07:14.479950 25804 ts_tablet_manager.cc:1918] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:14.480068 25804 log.cc:1196] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/a1a04e5bcefb45a181973e942650aa74
I20260810 23:07:14.480473 25804 ts_tablet_manager.cc:1939] T a1a04e5bcefb45a181973e942650aa74 P 1dfd0cf1cdca4b88a192d6fbaec5eed2: Deleting consensus metadata
I20260810 23:07:14.480947 25273 catalog_manager.cc:4841] TS 1a7c59aec6614b1fab4eceb4a7b60822 (127.14.75.3:40309): tablet a1a04e5bcefb45a181973e942650aa74 (table test-table [id=19d2ab0472b24376a3583498d7048894]) successfully deleted
I20260810 23:07:14.481714 25274 catalog_manager.cc:4841] TS 1dfd0cf1cdca4b88a192d6fbaec5eed2 (127.14.75.2:37899): tablet a1a04e5bcefb45a181973e942650aa74 (table test-table [id=19d2ab0472b24376a3583498d7048894]) successfully deleted
W20260810 23:07:14.482441 25273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:39357 (error 108)
I20260810 23:07:14.482709 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 25487
I20260810 23:07:14.483987 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:14.483710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39357 (local address 127.0.0.1:59634)
0810 23:07:14.483843 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:14.483914 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:07:14.484177 25273 catalog_manager.cc:4584] TS f3d1bb7dc6de4e7cb10934e526d53a52 (127.14.75.1:39357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1a04e5bcefb45a181973e942650aa74: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39357: connect: Connection refused (error 111)
I20260810 23:07:14.490397 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 25625
I20260810 23:07:14.497053 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 25257
2026-08-10T23:07:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2352 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25810](info): set up 2 sockets
Aug 10 23:07:14 dist-test-slave-srgv krb5kdc[25810](info): commencing operation
krb5kdc: starting...
W20260810 23:07:15.559305 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:07:15 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangeradmin_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangeradmin_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangerlookup_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangerlookup_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
I20260810 23:07:15.732082 14636 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmgqrPD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/postgres -l logfile start
2026-08-10 23:07:18.294 UTC [25842] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:07:18.294 UTC [25842] LOG: listening on IPv4 address "127.14.75.20", port 52769
2026-08-10 23:07:18.297 UTC [25842] LOG: listening on Unix socket "/tmp/.s.PGSQL.52769"
2026-08-10 23:07:18.335 UTC [25845] LOG: database system was shut down at 2026-08-10 23:07:16 UTC
2026-08-10 23:07:18.340 UTC [25842] LOG: database system is ready to accept connections
I20260810 23:07:19.307646 14636 mini_postgres.cc:96] Postgres bound to 52769
2026-08-10 23:07:19.315 UTC [25853] FATAL: database "slave" does not exist
127.14.75.20:52769 - accepting connections
I20260810 23:07:19.316396 14636 mini_ranger.cc:161] Starting Ranger...
I20260810 23:07:19.332841 14636 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:07:19.802480 14636 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:07:19.802587 14636 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin
2026-08-10 23:07:20,098 [I] DB FLAVOR :POSTGRES
2026-08-10 23:07:20,099 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:07:20,099 [I] Checking connection..
2026-08-10 23:07:20,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:07:20,377 [I] Checking connection passed.
2026-08-10 23:07:20,377 [I] --------- Verifying version history table ---------
2026-08-10 23:07:20,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:07:20,618 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:07:20,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:07:20,881 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:07:20,881 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:07:20,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:07:21.162 UTC [25919] WARNING: there is no transaction in progress
2026-08-10 23:07:21,170 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:07:21,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:07:21,432 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:07:21,432 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:07:21,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:07:21,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:07:21,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-10 23:07:22,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:07:22,426 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:07:22,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:07:22,683 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:07:22,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:07:22,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:07:23,190 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:07:23,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:07:24.165 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.182 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.199 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.214 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.231 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.249 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.315 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.321 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.336 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.346 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.356 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.370 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.380 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.387 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.394 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:24.400 UTC [26037] WARNING: there is no transaction in progress
2026-08-10 23:07:25,030 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:07:25,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:07:25,290 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:07:25,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:07:25,537 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:07:25,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:07:25,782 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:07:25,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:07:26,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:07:26,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-10 23:07:26,575 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:07:26,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:07:26,837 [I] DB_PATCHES have already been applied
I20260810 23:07:26.843380 14636 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmgqrPD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:07:26.843451 14636 mini_ranger.cc:193] Using host: 127.14.75.20
I20260810 23:07:26.847412 14636 mini_ranger.cc:227] Ranger admin URL: http://127.14.75.20:40671
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:07:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:07:28.007516 14636 mini_ranger.cc:160] Time spent starting Ranger: real 8.691s user 0.001s sys 0.011s
Aug 10, 2026 11:07:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:07:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:07:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:07:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:07:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:07:50.711364 14636 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangerkms_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/rangerkms_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.75.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.75.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:07:50 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403270, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:07:50.800254 14636 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:07:50.810829 14636 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:07:51.657559 14636 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:07:51.657657 14636 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms
I20260810 23:07:51.657768 14636 mini_ranger_kms.cc:227] Using postgres at 127.14.75.20:52769
2026-08-10 23:07:51,912 [I] DB FLAVOR :POSTGRES
2026-08-10 23:07:51,913 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:07:51,913 [I] Checking connection
2026-08-10 23:07:51,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:07:52,185 [I] connection success
2026-08-10 23:07:52,185 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:07:52,185 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:07:52,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:07:52,435 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:07:52,435 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:07:52,435 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:07:52,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:52769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:07:52,711 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:07:53.221155 14636 mini_ranger_kms.cc:310] Created kms service
I20260810 23:07:53.411298 14636 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:07:53.456991 14636 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:07:53 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403270, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:07:53.882082 14636 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:07:53.882238 14636 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:07:53.882263 14636 mini_ranger_kms.cc:240] Using host: 127.14.75.20
I20260810 23:07:53.886674 14636 mini_ranger_kms.cc:276] Ranger KMS PID: 26224
I20260810 23:07:53.886852 14636 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.75.20:47479
Aug 10, 2026 11:07:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:07:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:07:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:07:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:07:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:07:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 23:07:56.203943 14636 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.404s user 0.013s sys 0.224s
I20260810 23:07:56.204107 14636 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:07:56.204226 14636 mini_ranger_kms.cc:401] 127.14.75.20:47479/kms/v1/keys
Aug 10 23:07:56 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:07:56 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
Aug 10 23:07:57 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403276, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
Aug 10 23:07:59 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:07:59.649163 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42469
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42469
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:47479/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:59.771813 26279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:59.772123 26279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:59.772207 26279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:59.776520 26279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:59.776595 26279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:59.776616 26279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:59.776674 26279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:59.776705 26279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:59.782600 26279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.75.20:47479/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42469
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:07:59.783802 26279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:59.784921 26279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:59.794520 26279 server_base.cc:1034] running on GCE node
W20260810 23:07:59.794497 26291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:59.794458 26284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:59.794458 26293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:59.795480 26279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:59.796774 26279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:59.797971 26279 hybrid_clock.cc:648] HybridClock initialized: now 1786403279797955 us; error 36 us; skew 500 ppm
Aug 10 23:07:59 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403279, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:59.801362 26279 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:07:59.802385 26279 webserver.cc:460] Webserver started at http://127.14.75.62:35579/ using document root <none> and password file <none>
I20260810 23:07:59.803025 26279 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:59.803109 26279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:59.803339 26279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:07:59 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403279, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:07:59.856798 26279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "68b721146e044fdbaffdfad1f979053d"
format_stamp: "Formatted at 2026-08-10 23:07:59 on dist-test-slave-srgv"
server_key: "9cd724821970ef6ba25956f669b79011"
server_key_iv: "aae8d0985de01f41c19061258d7621da"
server_key_version: "kuduclusterkey@0"
I20260810 23:07:59.857736 26279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "68b721146e044fdbaffdfad1f979053d"
format_stamp: "Formatted at 2026-08-10 23:07:59 on dist-test-slave-srgv"
server_key: "9cd724821970ef6ba25956f669b79011"
server_key_iv: "aae8d0985de01f41c19061258d7621da"
server_key_version: "kuduclusterkey@0"
I20260810 23:07:59.863173 26279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:07:59.882012 26300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:59.883805 26279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:59.883972 26279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "68b721146e044fdbaffdfad1f979053d"
format_stamp: "Formatted at 2026-08-10 23:07:59 on dist-test-slave-srgv"
server_key: "9cd724821970ef6ba25956f669b79011"
server_key_iv: "aae8d0985de01f41c19061258d7621da"
server_key_version: "kuduclusterkey@0"
I20260810 23:07:59.884145 26279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:59.903256 26279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:59.907040 26279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:59.907274 26279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:59.917274 26279 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42469
I20260810 23:07:59.917302 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42469 every 8 connection(s)
I20260810 23:07:59.918699 26279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:59.922153 26354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:59.928288 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 26279
I20260810 23:07:59.928434 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:07:59.928498 26354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: Bootstrap starting.
I20260810 23:07:59.928771 14636 external_mini_cluster.cc:1428] Setting key b6fd0ea8335ac54188737cdc439dba3b
I20260810 23:07:59.931900 26354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:59.932766 26354 log.cc:824] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:59.935052 26354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: No bootstrap required, opened a new log
Aug 10 23:07:59 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:07:59.940012 26354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } }
I20260810 23:07:59.940364 26354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:59.940433 26354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b721146e044fdbaffdfad1f979053d, State: Initialized, Role: FOLLOWER
I20260810 23:07:59.940961 26354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } }
I20260810 23:07:59.941160 26354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:59.941247 26354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:59.941381 26354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:59.942659 26354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } }
I20260810 23:07:59.943049 26354 leader_election.cc:302] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b721146e044fdbaffdfad1f979053d; no voters:
I20260810 23:07:59.943491 26354 leader_election.cc:288] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:59.944778 26354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:59.945891 26359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:59.946303 26359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [term 1 LEADER]: Becoming Leader. State: Replica: 68b721146e044fdbaffdfad1f979053d, State: Running, Role: LEADER
I20260810 23:07:59.946846 26359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } }
I20260810 23:07:59.956866 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:59.930307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.14.75.62:42469)
0810 23:07:59.930845 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:59.930866 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:59.930912 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:07:59.931969 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:59.931996 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:59.932450 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:59.932835 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:59.933852 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:59.935398 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:59.936620 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:59.937349 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:59.937568 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:59.940854 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:59.940861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:59.940880 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:59.940919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:59.943173 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:59.945307 (+ 2134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:59.945313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:59.945320 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:59.945576 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:59.953399 (+ 7823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:59.953443 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:59.953445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:59.954225 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:59.954459 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:59.955445 (+ 986us) server_negotiation.cc:299] Negotiation successful
0810 23:07:59.955804 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":167,"threads_started":1}
I20260810 23:07:59.958729 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b721146e044fdbaffdfad1f979053d. Latest consensus state: current_term: 1 leader_uuid: "68b721146e044fdbaffdfad1f979053d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } } }
I20260810 23:07:59.959028 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:59.959360 26360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b721146e044fdbaffdfad1f979053d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b721146e044fdbaffdfad1f979053d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42469 } } }
I20260810 23:07:59.959550 26360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:59.961016 26354 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:07:59.962813 26374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:59.970242 26374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:59.978708 26374 catalog_manager.cc:1347] Generated new cluster ID: 5a54e1706a1e448ca00651f4b1f9058a
I20260810 23:07:59.978785 26374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:08:00.001251 26374 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:08:00.002920 26374 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:08:00.013165 26374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: Generated new TSK 0
I20260810 23:08:00.014290 26374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:08:00.761387 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:47479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:00.895186 26401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:00.895581 26401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:00.895690 26401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:00.900197 26401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:00.900296 26401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:00.900394 26401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:08:00.906508 26401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.75.20:47479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:00.908209 26401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:00.909653 26401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:00.922279 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:00.922667 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:00.922711 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:00.923969 26401 server_base.cc:1034] running on GCE node
I20260810 23:08:00.924508 26401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:00.925312 26401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:00.926573 26401 hybrid_clock.cc:648] HybridClock initialized: now 1786403280926539 us; error 47 us; skew 500 ppm
Aug 10 23:08:00 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:00.930522 26401 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:08:00.931699 26401 webserver.cc:460] Webserver started at http://127.14.75.1:46711/ using document root <none> and password file <none>
I20260810 23:08:00.932482 26401 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:00.932653 26401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:00.932915 26401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:00 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:00.950572 26401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "c1103cedd9b945aca19ff556f2fdd9af"
format_stamp: "Formatted at 2026-08-10 23:08:00 on dist-test-slave-srgv"
server_key: "13e96bfdae22ab782756a04a62058da6"
server_key_iv: "3b204c611ff23da21d45a06d891e8194"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:00.951502 26401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "c1103cedd9b945aca19ff556f2fdd9af"
format_stamp: "Formatted at 2026-08-10 23:08:00 on dist-test-slave-srgv"
server_key: "13e96bfdae22ab782756a04a62058da6"
server_key_iv: "3b204c611ff23da21d45a06d891e8194"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:00.957613 26401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260810 23:08:00.974767 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:00.976593 26401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:08:00.976761 26401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "c1103cedd9b945aca19ff556f2fdd9af"
format_stamp: "Formatted at 2026-08-10 23:08:00 on dist-test-slave-srgv"
server_key: "13e96bfdae22ab782756a04a62058da6"
server_key_iv: "3b204c611ff23da21d45a06d891e8194"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:00.976924 26401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:00.993335 26401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:00.996707 26401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:00.996932 26401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:00.997725 26401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:00.998708 26401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:00.998778 26401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:00.998859 26401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:00.998898 26401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.010594 26401 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:39569
I20260810 23:08:01.010614 26536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:39569 every 8 connection(s)
I20260810 23:08:01.011720 26401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:08:01.021114 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 26401
I20260810 23:08:01.021256 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:08:01.021601 14636 external_mini_cluster.cc:1428] Setting key 39c341d7840881520d7c8a60482fa78c
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403280, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:01.030627 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.013872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:48711 (local address 127.14.75.62:42469)
0810 23:08:01.014097 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.014102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.015019 (+ 917us) server_negotiation.cc:407] Connection header received
0810 23:08:01.016293 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.016297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.016363 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.016434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:01.018264 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.019156 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.020946 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.021322 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.021455 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.025757 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:01.025762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:01.025765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:01.025795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:01.027818 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.028519 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.028523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.028525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.028621 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.029086 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.029089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.029091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.029349 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:01.029467 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.030282 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.030435 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":57,"threads_started":1}
I20260810 23:08:01.032699 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.014364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42469 (local address 127.14.75.1:48711)
0810 23:08:01.014853 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.014909 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.016008 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.016881 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.016891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.017305 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:01.018045 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.018065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.019344 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.019348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.020779 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.020793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.021708 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.021713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.021919 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.023461 (+ 1542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:01.023484 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:01.025512 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:01.027971 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.027978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.027997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.028390 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.028740 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.028743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.028745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.028954 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.029556 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:01.029563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:01.029947 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.031124 (+ 1177us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.031539 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
I20260810 23:08:01.034577 26537 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42469
I20260810 23:08:01.034832 26537 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:01.035454 26537 heartbeater.cc:511] Master 127.14.75.62:42469 requested a full tablet report, sending...
I20260810 23:08:01.037395 26318 ts_manager.cc:194] Registered new tserver with Master: c1103cedd9b945aca19ff556f2fdd9af (127.14.75.1:39569)
I20260810 23:08:01.039143 26318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:48711
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:08:01.068624 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:47479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:08:01.232285 26545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:01.232707 26545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:01.232779 26545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:01.239162 26545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:01.239271 26545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:01.239369 26545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:08:01.247751 26545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.75.20:47479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:01.249472 26545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:01.250931 26545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:01.264094 26551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:01.264303 26545 server_base.cc:1034] running on GCE node
W20260810 23:08:01.264487 26554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:01.264782 26552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:01.265367 26545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:01.266256 26545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:01.267464 26545 hybrid_clock.cc:648] HybridClock initialized: now 1786403281267397 us; error 85 us; skew 500 ppm
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:01.271466 26545 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:08:01.272756 26545 webserver.cc:460] Webserver started at http://127.14.75.2:38607/ using document root <none> and password file <none>
I20260810 23:08:01.273533 26545 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:01.273715 26545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:01.274029 26545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:01.296959 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "4ede4cd17db0408caeac7e332fe5c6c3"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "e07ac4a8f88ef26f23d1a277b6cc3737"
server_key_iv: "09743a5c48ca68c66374da8d53f8a451"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.297813 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "4ede4cd17db0408caeac7e332fe5c6c3"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "e07ac4a8f88ef26f23d1a277b6cc3737"
server_key_iv: "09743a5c48ca68c66374da8d53f8a451"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.305061 26545 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.000s
I20260810 23:08:01.326896 26562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.328708 26545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:08:01.328959 26545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "4ede4cd17db0408caeac7e332fe5c6c3"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "e07ac4a8f88ef26f23d1a277b6cc3737"
server_key_iv: "09743a5c48ca68c66374da8d53f8a451"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.329219 26545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:01.361953 26545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:01.366182 26545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:01.366482 26545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:01.367458 26545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:01.368878 26545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:01.369024 26545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.369168 26545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:01.369393 26545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.386317 26545 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:35623
I20260810 23:08:01.389312 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 26545
I20260810 23:08:01.389477 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:08:01.388758 26545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:08:01.389793 14636 external_mini_cluster.cc:1428] Setting key ca50ee82d2a4d84509fb885d9ce61d1d
I20260810 23:08:01.389999 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:35623 every 8 connection(s)
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:08:01.418280 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.395910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:45393 (local address 127.14.75.62:42469)
0810 23:08:01.396108 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.396114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.397807 (+ 1693us) server_negotiation.cc:407] Connection header received
0810 23:08:01.399130 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.399133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.399234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.399361 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:01.400855 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.401741 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.408691 (+ 6950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.409104 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.409259 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.413591 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:01.413596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:01.413599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:01.413634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:01.415466 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.416210 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.416214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.416216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.416311 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.416764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.416768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.416769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.417066 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:01.417143 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.417951 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.418101 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:08:01.420244 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.394695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42469 (local address 127.14.75.2:45393)
0810 23:08:01.397620 (+ 2925us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.397690 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.398878 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.399550 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.399560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.399938 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:01.400683 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.400705 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.401906 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.401910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.403370 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.403382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.409624 (+ 6242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.409629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.409899 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.410720 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:01.410758 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:01.413292 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:01.415635 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.415656 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.415678 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.416072 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.416427 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.416429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.416431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.416650 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.417234 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:01.417242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:01.417648 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.418722 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.419109 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"thread_start_us":167,"threads_started":1}
I20260810 23:08:01.422259 26679 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42469
I20260810 23:08:01.422513 26679 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:01.423174 26679 heartbeater.cc:511] Master 127.14.75.62:42469 requested a full tablet report, sending...
I20260810 23:08:01.424506 26318 ts_manager.cc:194] Registered new tserver with Master: 4ede4cd17db0408caeac7e332fe5c6c3 (127.14.75.2:35623)
I20260810 23:08:01.425287 26318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:45393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:08:01.449512 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:47479/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
W20260810 23:08:01.625478 26684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:01.625937 26684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:01.626022 26684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:01.632162 26684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:01.632330 26684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:01.632453 26684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:08:01.640643 26684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.75.20:47479/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:01.642383 26684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:01.643824 26684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:01.664355 26690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:01.664361 26689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:01.665009 26684 server_base.cc:1034] running on GCE node
W20260810 23:08:01.669679 26692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:01.670472 26684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:01.671458 26684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:01.674762 26684 hybrid_clock.cc:648] HybridClock initialized: now 1786403281674663 us; error 107 us; skew 500 ppm
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:01.678929 26684 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:08:01.680243 26684 webserver.cc:460] Webserver started at http://127.14.75.3:32859/ using document root <none> and password file <none>
I20260810 23:08:01.681056 26684 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:01.681219 26684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:01.681604 26684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:01.715554 26684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "a5536ea753fb45d1b8b106119171c4c0"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "c55d4bb2283837d1c8d4bd315d505d4a"
server_key_iv: "ea7fd159a8047b9242ee516f1c483f90"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.716352 26684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "a5536ea753fb45d1b8b106119171c4c0"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "c55d4bb2283837d1c8d4bd315d505d4a"
server_key_iv: "ea7fd159a8047b9242ee516f1c483f90"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.722254 26684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 23:08:01.745775 26699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.747601 26684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:08:01.747767 26684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "a5536ea753fb45d1b8b106119171c4c0"
format_stamp: "Formatted at 2026-08-10 23:08:01 on dist-test-slave-srgv"
server_key: "c55d4bb2283837d1c8d4bd315d505d4a"
server_key_iv: "ea7fd159a8047b9242ee516f1c483f90"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:01.747913 26684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:01.770723 26684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:01.774884 26684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:01.775163 26684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:01.775951 26684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:01.777081 26684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:01.777180 26684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.777284 26684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:01.777340 26684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:01.790441 26684 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45341
I20260810 23:08:01.791839 26684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:08:01.796722 26815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45341 every 8 connection(s)
I20260810 23:08:01.799860 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 26684
I20260810 23:08:01.800000 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:08:01.800328 14636 external_mini_cluster.cc:1428] Setting key ef77619802121dfbe2fe971b777a7760
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:01.815310 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.798984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:58763 (local address 127.14.75.62:42469)
0810 23:08:01.799139 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.799144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.800928 (+ 1784us) server_negotiation.cc:407] Connection header received
0810 23:08:01.802218 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.802222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.802279 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.802333 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:01.804136 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.804988 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.806724 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.807054 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.807155 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.810543 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:01.810547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:01.810551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:01.810582 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:01.812463 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.813209 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.813212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.813214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.813315 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.813785 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.813788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.813790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.814046 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:01.814145 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.814994 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.815142 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:08:01.817313 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.799516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42469 (local address 127.14.75.3:58763)
0810 23:08:01.800768 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.800834 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.801965 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.802799 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.802809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.803234 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:01.803922 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.803941 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.805183 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.805186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.806577 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.806589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.807373 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.807376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.807637 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.808425 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:01.808446 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:01.810312 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:01.812643 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.812650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.812670 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.813062 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.813452 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:01.813455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:01.813457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:01.813670 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:01.814294 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:01.814301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:01.814698 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.815792 (+ 1094us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.816181 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":168,"threads_started":1}
I20260810 23:08:01.819025 26816 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42469
I20260810 23:08:01.819295 26816 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:01.819896 26816 heartbeater.cc:511] Master 127.14.75.62:42469 requested a full tablet report, sending...
I20260810 23:08:01.821130 26318 ts_manager.cc:194] Registered new tserver with Master: a5536ea753fb45d1b8b106119171c4c0 (127.14.75.3:45341)
I20260810 23:08:01.821918 26318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:58763
I20260810 23:08:01.826555 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:08:01.839980 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.829801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.14.75.62:42469)
0810 23:08:01.829922 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.829927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.830492 (+ 565us) server_negotiation.cc:407] Connection header received
0810 23:08:01.830679 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.830683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.830734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.830783 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:01.831804 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.832627 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.833851 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.834205 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.834299 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.836204 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:01.836208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:01.836211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:01.836242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:01.838014 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.838550 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.838553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.838555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.838646 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:01.838946 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:01.838949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:01.838951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:01.839216 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:01.839328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.839689 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.839817 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:08:01.844328 26318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:08:01.849570 26318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:08:01.885105 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.867007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45341 (local address 127.0.0.1:57120)
0810 23:08:01.872382 (+ 5375us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.872395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.872474 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.873263 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.873266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.873283 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:01.873378 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.873383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.879488 (+ 6105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.879491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.880686 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.880699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.884452 (+ 3753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.884457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.884611 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.884660 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.884780 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.884923 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5315,"thread_start_us":105,"threads_started":1}
I20260810 23:08:01.886397 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.866154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39569 (local address 127.0.0.1:57642)
0810 23:08:01.869657 (+ 3503us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.869671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.869784 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.871962 (+ 2178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.871969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.871994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:01.872164 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.872172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.876218 (+ 4046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.876221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.880925 (+ 4704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.880931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.885979 (+ 5048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.885983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.886077 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.886114 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.886184 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.886253 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3440,"thread_start_us":106,"threads_started":1}
I20260810 23:08:01.887502 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.867413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.14.75.3:45341)
0810 23:08:01.870074 (+ 2661us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.870097 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.872577 (+ 2480us) server_negotiation.cc:407] Connection header received
0810 23:08:01.872676 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.872681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.872921 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.873103 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:01.873573 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.879348 (+ 5775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.882007 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.884105 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.884220 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.884384 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.887218 (+ 2834us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.887350 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2549,"thread_start_us":88,"threads_started":1}
I20260810 23:08:01.888113 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.866582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.14.75.1:39569)
0810 23:08:01.869432 (+ 2850us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.869440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.870150 (+ 710us) server_negotiation.cc:407] Connection header received
0810 23:08:01.870262 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.870270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.871602 (+ 1332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.871770 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:01.874880 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.876055 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.881055 (+ 5000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.885649 (+ 4594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.885765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.885914 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.887807 (+ 1893us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.887938 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2708,"thread_start_us":93,"threads_started":1}
I20260810 23:08:01.888337 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.863711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35623 (local address 127.0.0.1:47474)
0810 23:08:01.868883 (+ 5172us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.868956 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.869103 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.874293 (+ 5190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.874297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.874314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:01.874418 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.874424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.875991 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.875994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.886404 (+ 10410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.886416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.887911 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.887915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:01.888013 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.888036 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:08:01.888111 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:08:01.888204 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4973,"mutex_wait_us":9088,"thread_start_us":127,"threads_started":1}
I20260810 23:08:01.889017 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.863830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.14.75.2:35623)
0810 23:08:01.873662 (+ 9832us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.873679 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.873702 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:08:01.873774 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.873778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.873966 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.874127 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:01.874566 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.875855 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.886614 (+ 10759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.887553 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.887692 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:01.887846 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:01.888771 (+ 925us) server_negotiation.cc:299] Negotiation successful
0810 23:08:01.888882 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9683,"thread_start_us":63,"threads_started":1}
I20260810 23:08:01.891973 26730 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 (DEFAULT_TABLE table=test-table [id=1df19e914db1427799fc4f9e58dbded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:01.891970 26594 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 (DEFAULT_TABLE table=test-table [id=1df19e914db1427799fc4f9e58dbded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:01.893286 26594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:01.894009 26730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:01.905850 26836 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Bootstrap starting.
I20260810 23:08:01.905870 26469 tablet_service.cc:1467] Processing CreateTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 (DEFAULT_TABLE table=test-table [id=1df19e914db1427799fc4f9e58dbded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:01.907140 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:01.909130 26836 tablet_bootstrap.cc:654] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:01.910089 26836 log.cc:824] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:01.918565 26835 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: Bootstrap starting.
I20260810 23:08:01.922188 26835 tablet_bootstrap.cc:654] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:01.923386 26835 log.cc:824] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:01.924413 26838 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Bootstrap starting.
I20260810 23:08:01.927379 26838 tablet_bootstrap.cc:654] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:01.928714 26838 log.cc:824] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:01.931152 26836 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: No bootstrap required, opened a new log
I20260810 23:08:01.931399 26836 ts_tablet_manager.cc:1397] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260810 23:08:01.939513 26836 raft_consensus.cc:348] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.939859 26836 raft_consensus.cc:374] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:01.939914 26836 raft_consensus.cc:729] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5536ea753fb45d1b8b106119171c4c0, State: Initialized, Role: FOLLOWER
I20260810 23:08:01.941151 26835 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: No bootstrap required, opened a new log
I20260810 23:08:01.941375 26835 ts_tablet_manager.cc:1397] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20260810 23:08:01.952973 26836 consensus_queue.cc:260] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.954550 26836 ts_tablet_manager.cc:1428] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Time spent starting tablet: real 0.022s user 0.002s sys 0.005s
I20260810 23:08:01.954674 26835 raft_consensus.cc:348] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.955006 26835 raft_consensus.cc:374] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:01.955014 26816 heartbeater.cc:503] Master 127.14.75.62:42469 was elected leader, sending a full tablet report...
I20260810 23:08:01.955078 26835 raft_consensus.cc:729] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ede4cd17db0408caeac7e332fe5c6c3, State: Initialized, Role: FOLLOWER
I20260810 23:08:01.955538 26835 consensus_queue.cc:260] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.956585 26835 ts_tablet_manager.cc:1428] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260810 23:08:01.958654 26679 heartbeater.cc:503] Master 127.14.75.62:42469 was elected leader, sending a full tablet report...
I20260810 23:08:01.969192 26838 tablet_bootstrap.cc:492] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: No bootstrap required, opened a new log
I20260810 23:08:01.969547 26838 ts_tablet_manager.cc:1397] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20260810 23:08:01.975904 26838 raft_consensus.cc:348] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.976423 26838 raft_consensus.cc:374] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:01.976543 26838 raft_consensus.cc:729] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1103cedd9b945aca19ff556f2fdd9af, State: Initialized, Role: FOLLOWER
I20260810 23:08:01.977239 26838 consensus_queue.cc:260] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.978786 26838 ts_tablet_manager.cc:1428] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:08:01.979257 26537 heartbeater.cc:503] Master 127.14.75.62:42469 was elected leader, sending a full tablet report...
I20260810 23:08:01.992971 26841 raft_consensus.cc:482] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:08:01.993285 26841 raft_consensus.cc:504] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:01.994755 26841 leader_election.cc:288] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ede4cd17db0408caeac7e332fe5c6c3 (127.14.75.2:35623), c1103cedd9b945aca19ff556f2fdd9af (127.14.75.1:39569)
Aug 10 23:08:01 dist-test-slave-srgv krb5kdc[25810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403281, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:02.001549 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.995059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39569 (local address 127.14.75.3:35811)
0810 23:08:01.996238 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:01.996256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:01.996361 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:01.997343 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:01.997347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:01.997368 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:01.997518 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:01.997526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.998626 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:01.998630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:02.000026 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:02.000039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.001062 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.001066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:02.001202 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:02.001227 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:08:02.001299 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:08:02.001379 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100}
I20260810 23:08:02.003209 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.996086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:35811 (local address 127.14.75.1:39569)
0810 23:08:01.996273 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.996278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:01.996315 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:08:01.996780 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:01.996783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:01.996837 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:01.996913 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:01.997661 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:01.998453 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.000194 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.000802 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.000909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:02.000983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:02.002980 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0810 23:08:02.003076 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:08:02.003854 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a68ab7d04ec785eb7dbd0dd5cde1" candidate_uuid: "a5536ea753fb45d1b8b106119171c4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1103cedd9b945aca19ff556f2fdd9af" is_pre_election: true
I20260810 23:08:02.004221 26471 raft_consensus.cc:2454] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5536ea753fb45d1b8b106119171c4c0 in term 0.
I20260810 23:08:02.005501 26702 leader_election.cc:302] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5536ea753fb45d1b8b106119171c4c0, c1103cedd9b945aca19ff556f2fdd9af; no voters:
I20260810 23:08:02.005865 26841 raft_consensus.cc:2781] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:08:02.005942 26841 raft_consensus.cc:482] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:08:02.005980 26841 raft_consensus.cc:3037] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:02.007354 26841 raft_consensus.cc:504] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:02.007740 26841 leader_election.cc:288] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 4ede4cd17db0408caeac7e332fe5c6c3 (127.14.75.2:35623), c1103cedd9b945aca19ff556f2fdd9af (127.14.75.1:39569)
I20260810 23:08:02.008459 26471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a68ab7d04ec785eb7dbd0dd5cde1" candidate_uuid: "a5536ea753fb45d1b8b106119171c4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1103cedd9b945aca19ff556f2fdd9af"
I20260810 23:08:02.008594 26471 raft_consensus.cc:3037] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:02.009979 26471 raft_consensus.cc:2454] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5536ea753fb45d1b8b106119171c4c0 in term 1.
I20260810 23:08:02.013705 26702 leader_election.cc:302] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5536ea753fb45d1b8b106119171c4c0, c1103cedd9b945aca19ff556f2fdd9af; no voters:
W20260810 23:08:02.013796 26538 tablet.cc:2369] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:08:02.013989 26841 raft_consensus.cc:2781] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:08:02.014222 26841 raft_consensus.cc:686] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 1 LEADER]: Becoming Leader. State: Replica: a5536ea753fb45d1b8b106119171c4c0, State: Running, Role: LEADER
I20260810 23:08:02.014600 26841 consensus_queue.cc:237] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } }
I20260810 23:08:02.024652 26318 catalog_manager.cc:5495] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5536ea753fb45d1b8b106119171c4c0 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "a5536ea753fb45d1b8b106119171c4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5536ea753fb45d1b8b106119171c4c0" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 } health_report { overall_health: UNKNOWN } } }
I20260810 23:08:02.030966 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.995182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35623 (local address 127.14.75.3:44267)
0810 23:08:02.001733 (+ 6551us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:02.001747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:02.001830 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:02.016776 (+ 14946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:02.016781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:02.016822 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:02.016946 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:02.016954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.021486 (+ 4532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.021490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:02.022661 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:02.022677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.030505 (+ 7828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.030510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:02.030613 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:02.030636 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:08:02.030699 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:08:02.030787 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6494,"thread_start_us":182,"threads_started":1}
I20260810 23:08:02.031512 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:01.995292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:44267 (local address 127.14.75.2:35623)
0810 23:08:01.995964 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:01.995969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:02.002229 (+ 6260us) server_negotiation.cc:407] Connection header received
0810 23:08:02.002291 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:02.002294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:02.002349 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:02.002420 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:02.020013 (+ 17593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.021221 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.029566 (+ 8345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.030226 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.030322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:02.030389 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:02.031301 (+ 912us) server_negotiation.cc:299] Negotiation successful
0810 23:08:02.031398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260810 23:08:02.032450 26632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a68ab7d04ec785eb7dbd0dd5cde1" candidate_uuid: "a5536ea753fb45d1b8b106119171c4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" is_pre_election: true
I20260810 23:08:02.032799 26632 raft_consensus.cc:2454] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5536ea753fb45d1b8b106119171c4c0 in term 0.
I20260810 23:08:02.033149 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d3a68ab7d04ec785eb7dbd0dd5cde1" candidate_uuid: "a5536ea753fb45d1b8b106119171c4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ede4cd17db0408caeac7e332fe5c6c3"
I20260810 23:08:02.033267 26631 raft_consensus.cc:3037] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:02.034754 26631 raft_consensus.cc:2454] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5536ea753fb45d1b8b106119171c4c0 in term 1.
W20260810 23:08:02.060200 26817 tablet.cc:2369] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:08:02.076700 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:02.071575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.14.75.3:45341)
0810 23:08:02.071707 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:02.071711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:02.071725 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:08:02.071814 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:02.071817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:02.071886 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:02.071995 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:08:02.072431 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.073231 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.074547 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:02.074881 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:02.074993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:02.075337 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:08:02.075917 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:08:02.076044 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:02.076438 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 23:08:02.076508 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:08:02.086158 26471 raft_consensus.cc:1261] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P c1103cedd9b945aca19ff556f2fdd9af [term 1 FOLLOWER]: Refusing update from remote peer a5536ea753fb45d1b8b106119171c4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:02.087177 26846 consensus_queue.cc:1035] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1103cedd9b945aca19ff556f2fdd9af" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 39569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:02.095525 26631 raft_consensus.cc:1261] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P 4ede4cd17db0408caeac7e332fe5c6c3 [term 1 FOLLOWER]: Refusing update from remote peer a5536ea753fb45d1b8b106119171c4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:02.096321 26846 consensus_queue.cc:1035] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ede4cd17db0408caeac7e332fe5c6c3" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:02.116845 26318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:08:02.117040 26318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:08:02.121804 26318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68b721146e044fdbaffdfad1f979053d: Sending DeleteTablet for 3 replicas of tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1
I20260810 23:08:02.123322 26730 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:57120
I20260810 23:08:02.123719 26594 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:47474
I20260810 23:08:02.124238 26469 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:57642
I20260810 23:08:02.124786 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 26401
W20260810 23:08:02.136370 26303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:39569 (error 108)
I20260810 23:08:02.136592 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 26545
I20260810 23:08:02.146286 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:02.145876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:39569 (local address 127.0.0.1:57658)
0810 23:08:02.146087 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:02.146178 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260810 23:08:02.146675 26303 catalog_manager.cc:4584] TS c1103cedd9b945aca19ff556f2fdd9af (127.14.75.1:39569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0d3a68ab7d04ec785eb7dbd0dd5cde1: Network error: Client connection negotiation failed: client connection to 127.14.75.1:39569: connect: Connection refused (error 111)
I20260810 23:08:02.153005 26859 tablet_replica.cc:330] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: stopping tablet replica
I20260810 23:08:02.153563 26859 raft_consensus.cc:2229] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:08:02.154182 26859 raft_consensus.cc:2258] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:08:02.156175 26831 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:08:02.155567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35623 (local address 127.0.0.1:47484)
0810 23:08:02.155760 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:02.155775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:02.155872 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:02.156058 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.2:35623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 23:08:02.156236 26859 ts_tablet_manager.cc:1905] T b0d3a68ab7d04ec785eb7dbd0dd5cde1 P a5536ea753fb45d1b8b106119171c4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:08:02.156957 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 26684
W20260810 23:08:02.171958 26831 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:08:02.171359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45341 (local address 127.0.0.1:57126)
0810 23:08:02.171541 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:02.171557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:02.171678 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:02.171845 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.3:45341: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 23:08:02.172804 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 26279
I20260810 23:08:02.181861 14636 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:08:02.527230 14636 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:08:02.527374 14636 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:08:02.872743 14636 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:08:02.873 UTC [25842] LOG: received smart shutdown request
2026-08-10 23:08:02.877 UTC [25842] LOG: background worker "logical replication launcher" (PID 25851) exited with exit code 1
2026-08-10 23:08:02.877 UTC [25846] LOG: shutting down
2026-08-10 23:08:03.063 UTC [25842] LOG: database system is shut down
2026-08-10T23:08:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48688 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:08:03 dist-test-slave-srgv krb5kdc[26873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:08:03 dist-test-slave-srgv krb5kdc[26873](info): set up 2 sockets
Aug 10 23:08:03 dist-test-slave-srgv krb5kdc[26873](info): commencing operation
krb5kdc: starting...
W20260810 23:08:04.260618 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:08:04 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:08:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:08:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangeradmin_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangeradmin_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangerlookup_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangerlookup_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
I20260810 23:08:04.443804 14636 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmgqrPD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/postgres -l logfile start
2026-08-10 23:08:07.017 UTC [26903] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:08:07.017 UTC [26903] LOG: listening on IPv4 address "127.14.75.20", port 40353
2026-08-10 23:08:07.022 UTC [26903] LOG: listening on Unix socket "/tmp/.s.PGSQL.40353"
2026-08-10 23:08:07.052 UTC [26906] LOG: database system was shut down at 2026-08-10 23:08:05 UTC
2026-08-10 23:08:07.057 UTC [26903] LOG: database system is ready to accept connections
I20260810 23:08:08.079643 14636 mini_postgres.cc:96] Postgres bound to 40353
2026-08-10 23:08:08.086 UTC [26915] FATAL: database "slave" does not exist
127.14.75.20:40353 - accepting connections
I20260810 23:08:08.087291 14636 mini_ranger.cc:161] Starting Ranger...
I20260810 23:08:08.103314 14636 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:08:08.578997 14636 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:08:08.579090 14636 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin
2026-08-10 23:08:08,868 [I] DB FLAVOR :POSTGRES
2026-08-10 23:08:08,869 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:08:08,869 [I] Checking connection..
2026-08-10 23:08:08,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:08:09,117 [I] Checking connection passed.
2026-08-10 23:08:09,118 [I] --------- Verifying version history table ---------
2026-08-10 23:08:09,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:08:09,373 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:08:09,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:08:09,613 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:08:09,613 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:08:09,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:08:09.876 UTC [26981] WARNING: there is no transaction in progress
2026-08-10 23:08:09,883 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:08:09,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:08:10,152 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:08:10,153 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:08:10,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:08:10,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:08:10,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-srgv.c.gcp-upstream.internal','N') ;"
2026-08-10 23:08:10,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:08:11,183 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:08:11,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:08:11,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:08:11,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:08:11,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:08:11,926 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:08:11,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:08:12.938 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:12.955 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:12.972 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:12.989 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.005 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.023 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.079 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.085 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.100 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.111 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.121 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.136 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.145 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.152 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.159 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13.166 UTC [27093] WARNING: there is no transaction in progress
2026-08-10 23:08:13,833 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:08:13,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:08:14,087 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:08:14,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:08:14,329 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:08:14,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:08:14,582 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:08:14,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:08:14,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:08:15,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-srgv.c.gcp-upstream.internal';"
2026-08-10 23:08:15,312 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:08:15,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:08:15,574 [I] DB_PATCHES have already been applied
I20260810 23:08:15.580715 14636 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmgqrPD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:08:15.580793 14636 mini_ranger.cc:193] Using host: 127.14.75.20
I20260810 23:08:15.584914 14636 mini_ranger.cc:227] Ranger admin URL: http://127.14.75.20:44329
Aug 10, 2026 11:08:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:08:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:08:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:08:16.726941 14636 mini_ranger.cc:160] Time spent starting Ranger: real 8.640s user 0.000s sys 0.011s
Aug 10, 2026 11:08:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:08:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:08:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:08:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:08:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:08:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:08:39.640075 14636 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.75.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.75.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangerkms_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.75.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/rangerkms_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.75.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.75.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.75.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/HTTP_127.14.75.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:08:39 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403319, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:08:39.742480 14636 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:08:39.753010 14636 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:08:40.675506 14636 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:08:40.675598 14636 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms
I20260810 23:08:40.675617 14636 mini_ranger_kms.cc:227] Using postgres at 127.14.75.20:40353
2026-08-10 23:08:40,870 [I] DB FLAVOR :POSTGRES
2026-08-10 23:08:40,870 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:08:40,870 [I] Checking connection
2026-08-10 23:08:40,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:08:41,164 [I] connection success
2026-08-10 23:08:41,164 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:08:41,164 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:08:41,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:08:41,418 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:08:41,419 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:08:41,419 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:08:41,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.75.20:40353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:08:41,701 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:08:42.210703 14636 mini_ranger_kms.cc:310] Created kms service
I20260810 23:08:42.392167 14636 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:08:42.437880 14636 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:08:42 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403319, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:42.904304 14636 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:08:42.904471 14636 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmgqrPD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmgqrPD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:08:42.904495 14636 mini_ranger_kms.cc:240] Using host: 127.14.75.20
I20260810 23:08:42.908707 14636 mini_ranger_kms.cc:276] Ranger KMS PID: 27281
I20260810 23:08:42.908798 14636 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.75.20:51565
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:08:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 23:08:45.275492 14636 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.533s user 0.007s sys 0.171s
I20260810 23:08:45.275658 14636 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:08:45.275765 14636 mini_ranger_kms.cc:401] 127.14.75.20:51565/kms/v1/keys
Aug 10 23:08:45 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403325, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:08:46 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403325, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
Aug 10 23:08:46 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403325, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.75.20@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
Aug 10 23:08:48 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:08:49.041844 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38101
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:49.179065 27332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:49.179416 27332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:49.179498 27332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:49.183771 27332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:49.183878 27332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:49.183925 27332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:49.183987 27332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:49.184052 27332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:49.190109 27332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.75.20:51565/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38101
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:49.191342 27332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:49.192443 27332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:08:49.202947 27332 server_base.cc:1034] running on GCE node
W20260810 23:08:49.203104 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:49.202973 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:49.203311 27340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:49.203941 27332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:49.205255 27332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:49.206476 27332 hybrid_clock.cc:648] HybridClock initialized: now 1786403329206462 us; error 35 us; skew 500 ppm
Aug 10 23:08:49 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403329, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:49.212989 27332 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:08:49.214221 27332 webserver.cc:460] Webserver started at http://127.14.75.62:35135/ using document root <none> and password file <none>
I20260810 23:08:49.214897 27332 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:49.214962 27332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:49.215214 27332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:49 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403329, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:49.280228 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "efa564672e3140078c95f429f044285e"
format_stamp: "Formatted at 2026-08-10 23:08:49 on dist-test-slave-srgv"
server_key: "e1f74b25eada9083819c022b73b1a14c"
server_key_iv: "6d8dccdd1b03b1bf3656c16c5a6fba01"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:49.281002 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "efa564672e3140078c95f429f044285e"
format_stamp: "Formatted at 2026-08-10 23:08:49 on dist-test-slave-srgv"
server_key: "e1f74b25eada9083819c022b73b1a14c"
server_key_iv: "6d8dccdd1b03b1bf3656c16c5a6fba01"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:49.286245 27332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:08:49.304612 27347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:49.306363 27332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:08:49.306521 27332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "efa564672e3140078c95f429f044285e"
format_stamp: "Formatted at 2026-08-10 23:08:49 on dist-test-slave-srgv"
server_key: "e1f74b25eada9083819c022b73b1a14c"
server_key_iv: "6d8dccdd1b03b1bf3656c16c5a6fba01"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:49.306676 27332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:49.331024 27332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:49.335011 27332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:49.335237 27332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:49.344864 27332 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:38101
I20260810 23:08:49.344873 27400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:38101 every 8 connection(s)
I20260810 23:08:49.346163 27332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:08:49.349202 27401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:49.352255 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27332
I20260810 23:08:49.352425 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:08:49.352768 14636 external_mini_cluster.cc:1428] Setting key cbdd610fc0f0baa9abb62801599b8b66
I20260810 23:08:49.355844 27401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Bootstrap starting.
I20260810 23:08:49.358541 27401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:49.359387 27401 log.cc:824] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:49.361709 27401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: No bootstrap required, opened a new log
Aug 10 23:08:49 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:49.366266 27401 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:49.366569 27401 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:49.366640 27401 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa564672e3140078c95f429f044285e, State: Initialized, Role: FOLLOWER
I20260810 23:08:49.367148 27401 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:49.367319 27401 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:08:49.367381 27401 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:08:49.367467 27401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:49.368525 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:49.354382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.14.75.62:38101)
0810 23:08:49.354904 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:49.354924 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:49.354969 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:08:49.355795 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:49.355818 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:49.356256 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:49.356626 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:49.357655 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:49.359135 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:49.360243 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:49.360727 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:49.360927 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:49.363616 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:49.363629 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:49.363644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:49.363670 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:49.365455 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:49.365943 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:49.365948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:49.365953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:49.366065 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:49.366360 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:49.366364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:49.366366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:49.366928 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:49.367156 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:49.367435 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:08:49.367743 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":150,"threads_started":1}
I20260810 23:08:49.368825 27401 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:49.369220 27401 leader_election.cc:302] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa564672e3140078c95f429f044285e; no voters:
I20260810 23:08:49.369611 27401 leader_election.cc:288] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:08:49.369976 27407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:08:49.370168 27407 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 LEADER]: Becoming Leader. State: Replica: efa564672e3140078c95f429f044285e, State: Running, Role: LEADER
I20260810 23:08:49.370465 27407 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:49.370913 27401 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:08:49.372206 27407 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efa564672e3140078c95f429f044285e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:08:49.373148 27401 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:08:49.374261 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa564672e3140078c95f429f044285e. Latest consensus state: current_term: 1 leader_uuid: "efa564672e3140078c95f429f044285e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:08:49.374373 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:49.374356 27407 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:49.378698 27421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:08:49.383153 27421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:08:49.391135 27421 catalog_manager.cc:1347] Generated new cluster ID: e1122d4d84484a9bb136b7dcabced8fe
I20260810 23:08:49.391311 27421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:08:49.411263 27421 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:08:49.412909 27421 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:08:49.418821 27421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Generated new TSK 0
I20260810 23:08:49.419835 27421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:08:50.233000 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:50.398921 27446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:50.399410 27446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:50.399494 27446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:50.405249 27446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:50.405349 27446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:50.405465 27446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:08:50.413527 27446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:50.415129 27446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:50.416534 27446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:50.431403 27454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:50.431375 27453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:50.432057 27456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:50.433954 27446 server_base.cc:1034] running on GCE node
I20260810 23:08:50.434657 27446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:50.435631 27446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:50.436954 27446 hybrid_clock.cc:648] HybridClock initialized: now 1786403330436869 us; error 107 us; skew 500 ppm
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:50.441228 27446 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:08:50.442550 27446 webserver.cc:460] Webserver started at http://127.14.75.1:39801/ using document root <none> and password file <none>
I20260810 23:08:50.443348 27446 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:50.443457 27446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:50.443722 27446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:50.464697 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "1c6b8c88147749968c3e6f641479f910"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "0de39d76f7fa4e1d35160a9c0a6e89f1"
server_key_iv: "53bec0982c813587e9aacc6f53c98fd5"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.468542 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "1c6b8c88147749968c3e6f641479f910"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "0de39d76f7fa4e1d35160a9c0a6e89f1"
server_key_iv: "53bec0982c813587e9aacc6f53c98fd5"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.474639 27446 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 23:08:50.498662 27464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.500260 27446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:08:50.500437 27446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "1c6b8c88147749968c3e6f641479f910"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "0de39d76f7fa4e1d35160a9c0a6e89f1"
server_key_iv: "53bec0982c813587e9aacc6f53c98fd5"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.500591 27446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:50.539757 27446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:50.543911 27446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:50.544241 27446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:50.545112 27446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:50.546298 27446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:50.546412 27446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.546542 27446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:50.546630 27446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.560636 27446 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:34429
I20260810 23:08:50.562124 27446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:08:50.569571 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27446
I20260810 23:08:50.569725 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:08:50.570071 14636 external_mini_cluster.cc:1428] Setting key 27c9b75cddd064371f3c20b62044a3db
I20260810 23:08:50.570317 27583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:34429 every 8 connection(s)
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:50.597118 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:50.565660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50723 (local address 127.14.75.62:38101)
0810 23:08:50.577632 (+ 11972us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:50.577637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:50.577658 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:08:50.577709 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:50.577712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:50.577767 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:50.577881 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:50.583221 (+ 5340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.584112 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.586234 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.586632 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.586715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:50.590213 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:50.590218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:50.590221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:50.590252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:50.592217 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:50.592947 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:50.592951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:50.592953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:50.593049 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:50.595588 (+ 2539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:50.595593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:50.595595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:50.595864 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:50.595956 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:50.596777 (+ 821us) server_negotiation.cc:299] Negotiation successful
0810 23:08:50.596940 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11862,"thread_start_us":83,"threads_started":1}
I20260810 23:08:50.599212 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:50.566339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.1:50723)
0810 23:08:50.573801 (+ 7462us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:50.573886 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:50.574999 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:50.581790 (+ 6791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:50.581804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:50.582227 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:50.582974 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:50.582996 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.584306 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.584310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:50.586036 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:50.586054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.586967 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.586970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:50.587202 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:50.588003 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:50.588026 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:50.589968 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:50.592390 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:50.592396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:50.592416 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:50.592818 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:50.593161 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:50.593164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:50.593166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:50.593356 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:50.596041 (+ 2685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:50.596049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:50.596456 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 23:08:50.597681 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0810 23:08:50.598097 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7158,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:08:50.606551 27584 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:50.606832 27584 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:50.607496 27584 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:50.609543 27358 ts_manager.cc:194] Registered new tserver with Master: 1c6b8c88147749968c3e6f641479f910 (127.14.75.1:34429)
I20260810 23:08:50.611089 27358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:50723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:08:50.637848 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:08:50.776815 27590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:50.777160 27590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:50.777289 27590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:50.781710 27590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:50.781804 27590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:50.781917 27590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:08:50.787829 27590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:50.789156 27590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:50.790676 27590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:50.803272 27597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:50.803318 27599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:50.803637 27590 server_base.cc:1034] running on GCE node
W20260810 23:08:50.803692 27596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:50.804141 27590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:50.804924 27590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:50.806214 27590 hybrid_clock.cc:648] HybridClock initialized: now 1786403330806185 us; error 78 us; skew 500 ppm
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:50.810078 27590 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:08:50.811443 27590 webserver.cc:460] Webserver started at http://127.14.75.2:38285/ using document root <none> and password file <none>
I20260810 23:08:50.812254 27590 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:50.812354 27590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:50.812719 27590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:50.835238 27590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "7870b63237bc490384102c704c7f1274"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "34ddff37abc7a44076c0771cdd395b70"
server_key_iv: "cc278e2ef08c2c7a936fa94b11245fea"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.836198 27590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "7870b63237bc490384102c704c7f1274"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "34ddff37abc7a44076c0771cdd395b70"
server_key_iv: "cc278e2ef08c2c7a936fa94b11245fea"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.842306 27590 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 23:08:50.862929 27606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.864806 27590 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 23:08:50.864961 27590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "7870b63237bc490384102c704c7f1274"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "34ddff37abc7a44076c0771cdd395b70"
server_key_iv: "cc278e2ef08c2c7a936fa94b11245fea"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:50.865118 27590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:50.894037 27590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:50.898717 27590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:50.899008 27590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:50.899859 27590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:50.901044 27590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:50.901173 27590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.901288 27590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:50.901363 27590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:50.914844 27590 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33103
I20260810 23:08:50.914873 27725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33103 every 8 connection(s)
I20260810 23:08:50.915987 27590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:08:50.922828 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27590
I20260810 23:08:50.922996 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:08:50.923332 14636 external_mini_cluster.cc:1428] Setting key 1ef7d51d81ed8e6a5cea5d36f713715a
Aug 10 23:08:50 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:08:50.944290 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:50.919580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.2:50003)
0810 23:08:50.920183 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:50.920251 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:50.921378 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:50.922220 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:50.922229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:50.922608 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:50.923590 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:50.923613 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.924838 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.924842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:50.926332 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:50.926344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.927231 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.927235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:50.927454 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:50.928276 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:50.928298 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:50.937934 (+ 9636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:50.940526 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:50.940535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:50.940559 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:50.941000 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:50.941379 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:50.941382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:50.941384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:50.941640 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:50.942160 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:50.942167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:50.942573 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 23:08:50.942779 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 23:08:50.943145 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":160,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:08:50.945111 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:50.919031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:50003 (local address 127.14.75.62:38101)
0810 23:08:50.919177 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:50.919181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:50.920369 (+ 1188us) server_negotiation.cc:407] Connection header received
0810 23:08:50.921675 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:50.921679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:50.921751 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:50.921807 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:50.923802 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.924646 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.926493 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:50.926913 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:50.927002 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:50.938272 (+ 11270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:50.938281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:50.938286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:50.938336 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:50.940333 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:50.941150 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:50.941153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:50.941156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:50.941269 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:50.941759 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:50.941762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:50.941764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:50.942006 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:50.942099 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:50.944807 (+ 2708us) server_negotiation.cc:299] Negotiation successful
0810 23:08:50.944964 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:08:50.946193 27726 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:50.946451 27726 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:50.947013 27726 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:50.948277 27358 ts_manager.cc:194] Registered new tserver with Master: 7870b63237bc490384102c704c7f1274 (127.14.75.2:33103)
I20260810 23:08:50.948978 27358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:50003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:08:50.965116 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:51.152012 27731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:51.152875 27731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:51.153034 27731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:51.159402 27731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:51.159567 27731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:51.159703 27731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:08:51.168080 27731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:51.169831 27731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:51.171620 27731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:51.185806 27738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:51.186108 27731 server_base.cc:1034] running on GCE node
W20260810 23:08:51.185854 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:51.186414 27740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:51.186968 27731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:51.187822 27731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:51.204532 27731 hybrid_clock.cc:648] HybridClock initialized: now 1786403331204417 us; error 146 us; skew 500 ppm
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:51.208822 27731 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:08:51.210060 27731 webserver.cc:460] Webserver started at http://127.14.75.3:41333/ using document root <none> and password file <none>
I20260810 23:08:51.210876 27731 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:51.211030 27731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:51.211403 27731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403330, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:51.241575 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "290539aa34cf41d5b908b6912a22831e"
format_stamp: "Formatted at 2026-08-10 23:08:51 on dist-test-slave-srgv"
server_key: "465abe7c73d3615dfaf404ebd12af664"
server_key_iv: "f1ee547db8b3b080c27e4c7aa6079516"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:51.242609 27731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "290539aa34cf41d5b908b6912a22831e"
format_stamp: "Formatted at 2026-08-10 23:08:51 on dist-test-slave-srgv"
server_key: "465abe7c73d3615dfaf404ebd12af664"
server_key_iv: "f1ee547db8b3b080c27e4c7aa6079516"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:51.252143 27731 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260810 23:08:51.274223 27747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.276104 27731 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 23:08:51.276273 27731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "290539aa34cf41d5b908b6912a22831e"
format_stamp: "Formatted at 2026-08-10 23:08:51 on dist-test-slave-srgv"
server_key: "465abe7c73d3615dfaf404ebd12af664"
server_key_iv: "f1ee547db8b3b080c27e4c7aa6079516"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:51.276433 27731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:51.310552 27731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:51.314625 27731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:51.314903 27731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:51.315686 27731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:51.316787 27731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:51.316900 27731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.317001 27731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:51.317080 27731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.329321 27731 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:44383
I20260810 23:08:51.330687 27731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:08:51.332234 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27731
I20260810 23:08:51.332365 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:08:51.332655 14636 external_mini_cluster.cc:1428] Setting key 6c70945659f94b77d0de2ec1fb00dc4e
I20260810 23:08:51.336817 27863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:44383 every 8 connection(s)
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:51.357745 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:51.340368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:39183 (local address 127.14.75.62:38101)
0810 23:08:51.340493 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:51.340498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:51.341843 (+ 1345us) server_negotiation.cc:407] Connection header received
0810 23:08:51.343124 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:51.343129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:51.343187 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:51.343242 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:51.345103 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.345950 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.349396 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.349790 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.349862 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:51.353226 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:51.353230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:51.353233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:51.353263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:51.355154 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:51.355817 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:51.355821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:51.355823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:51.355915 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:51.356337 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:51.356341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:51.356343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:51.356566 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:51.356637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:51.357370 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 23:08:51.357555 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:08:51.361439 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:51.339202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.3:39183)
0810 23:08:51.341678 (+ 2476us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:51.341738 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:51.342851 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:51.343719 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:51.343729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:51.344182 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:51.344918 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:51.344939 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.347733 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.347736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:51.349254 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:51.349267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.350093 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.350097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:51.350304 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:51.351030 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:51.351052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:51.353009 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:51.355312 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:51.355318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:51.355337 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:51.355695 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:51.356019 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:51.356022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:51.356023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:51.356230 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:51.356715 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:51.356722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:51.357080 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 23:08:51.359891 (+ 2811us) client_negotiation.cc:240] Negotiation successful
0810 23:08:51.360313 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2234,"thread_start_us":177,"threads_started":1}
I20260810 23:08:51.362965 27864 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:51.363229 27864 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:51.363809 27864 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:51.365043 27358 ts_manager.cc:194] Registered new tserver with Master: 290539aa34cf41d5b908b6912a22831e (127.14.75.3:44383)
I20260810 23:08:51.365823 27358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:39183
I20260810 23:08:51.371909 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:08:51.372891 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27446
I20260810 23:08:51.381742 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27590
I20260810 23:08:51.393904 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27731
I20260810 23:08:51.403846 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27332
I20260810 23:08:51.412650 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--webserver_interface=127.14.75.62
--webserver_port=35135
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38101
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:51.532341 27867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:51.532709 27867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:51.532788 27867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:51.537106 27867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:51.537230 27867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:51.537267 27867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:51.537307 27867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:51.537344 27867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:51.543300 27867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38101
--ranger_config_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=35135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:51.544639 27867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:51.545737 27867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:51.554827 27875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:51.554785 27873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:51.554818 27872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:51.555234 27867 server_base.cc:1034] running on GCE node
I20260810 23:08:51.555893 27867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:51.557060 27867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:51.558243 27867 hybrid_clock.cc:648] HybridClock initialized: now 1786403331558230 us; error 26 us; skew 500 ppm
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:51.561301 27867 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:08:51.562337 27867 webserver.cc:460] Webserver started at http://127.14.75.62:35135/ using document root <none> and password file <none>
I20260810 23:08:51.562863 27867 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:51.562932 27867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:51.649726 27867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:08:51.652871 27883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.654855 27867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:08:51.655011 27867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "efa564672e3140078c95f429f044285e"
format_stamp: "Formatted at 2026-08-10 23:08:49 on dist-test-slave-srgv"
server_key: "e1f74b25eada9083819c022b73b1a14c"
server_key_iv: "6d8dccdd1b03b1bf3656c16c5a6fba01"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:51.655898 27867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:51.674959 27867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:51.679016 27867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:51.679232 27867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:51.690258 27867 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:38101
I20260810 23:08:51.690284 27937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:38101 every 8 connection(s)
I20260810 23:08:51.691972 27867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:08:51.692829 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27867
I20260810 23:08:51.693442 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:34429
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:38101
--webserver_port=39801
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:08:51.697933 27938 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:08:51.700665 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Bootstrap starting.
I20260810 23:08:51.724506 27938 log.cc:824] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:51.733217 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:08:51.733778 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Bootstrap complete.
I20260810 23:08:51.739539 27938 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:51.740200 27938 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa564672e3140078c95f429f044285e, State: Initialized, Role: FOLLOWER
I20260810 23:08:51.740808 27938 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:51.740962 27938 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:08:51.741039 27938 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:08:51.741139 27938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:08:51.743880 27938 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:51.744279 27938 leader_election.cc:302] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa564672e3140078c95f429f044285e; no voters:
I20260810 23:08:51.744647 27938 leader_election.cc:288] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:08:51.744791 27943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:08:51.745072 27943 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [term 2 LEADER]: Becoming Leader. State: Replica: efa564672e3140078c95f429f044285e, State: Running, Role: LEADER
I20260810 23:08:51.745436 27943 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } }
I20260810 23:08:51.746034 27938 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:08:51.746789 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efa564672e3140078c95f429f044285e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:08:51.746920 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:51.747253 27946 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa564672e3140078c95f429f044285e. Latest consensus state: current_term: 2 leader_uuid: "efa564672e3140078c95f429f044285e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa564672e3140078c95f429f044285e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38101 } } }
I20260810 23:08:51.747344 27946 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:51.747898 27953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:08:51.748410 27938 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:08:51.755419 27953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:08:51.756142 27953 catalog_manager.cc:1259] Loaded cluster ID: e1122d4d84484a9bb136b7dcabced8fe
I20260810 23:08:51.756220 27953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:08:51.758850 27953 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:08:51.759598 27953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Loaded TSK: 0
I20260810 23:08:51.761123 27953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:08:51.865437 27941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:51.865839 27941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:51.865914 27941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:51.872670 27941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:51.872778 27941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:51.872875 27941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:08:51.881558 27941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=39801
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:51.883049 27941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:51.884496 27941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:51.898670 27977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:51.898799 27941 server_base.cc:1034] running on GCE node
W20260810 23:08:51.899010 27979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:51.899362 27981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:51.899886 27941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:51.900727 27941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:51.901952 27941 hybrid_clock.cc:648] HybridClock initialized: now 1786403331901928 us; error 40 us; skew 500 ppm
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:51.905866 27941 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:08:51.907145 27941 webserver.cc:460] Webserver started at http://127.14.75.1:39801/ using document root <none> and password file <none>
I20260810 23:08:51.907971 27941 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:51.908083 27941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:51.934155 27941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:08:51.937222 27990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.938728 27941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:08:51.938889 27941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "1c6b8c88147749968c3e6f641479f910"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "0de39d76f7fa4e1d35160a9c0a6e89f1"
server_key_iv: "53bec0982c813587e9aacc6f53c98fd5"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:51.939496 27941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:51.957608 27941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:51.961656 27941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:51.961916 27941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:51.962611 27941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:51.963681 27941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:51.963735 27941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.963804 27941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:51.963835 27941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:51.975574 27941 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:34429
I20260810 23:08:51.976931 27941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:08:51.981838 28107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:34429 every 8 connection(s)
I20260810 23:08:51.983907 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 27941
I20260810 23:08:51.984355 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:33103
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:38101
--webserver_port=38285
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:51 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403331, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:52.007423 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:51.985730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:59743 (local address 127.14.75.62:38101)
0810 23:08:51.986295 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:51.986326 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:51.987340 (+ 1014us) server_negotiation.cc:407] Connection header received
0810 23:08:51.988928 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:51.988944 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:51.989377 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:51.989793 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:51.991654 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.993271 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.995065 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.995814 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.995976 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:51.999582 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:51.999594 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:51.999615 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:51.999660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:52.002351 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.003219 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.003228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.003235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.003418 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.003929 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.003933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.003935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.004590 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:52.004735 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:52.005693 (+ 958us) server_negotiation.cc:299] Negotiation successful
0810 23:08:52.006097 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":140,"threads_started":1}
I20260810 23:08:52.009271 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:51.986542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.1:59743)
0810 23:08:51.987123 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:51.987197 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:51.988378 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:51.990361 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:51.990370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:51.990768 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:51.991433 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:51.991455 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.993474 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.993477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:51.994905 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:51.994917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:51.996272 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:51.996276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:51.996468 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:51.997259 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:51.997287 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:51.999269 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:52.002543 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.002555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.002576 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.003029 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.003562 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.003568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.003572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.003789 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.004821 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:52.004832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:52.005236 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 23:08:52.007835 (+ 2599us) client_negotiation.cc:240] Negotiation successful
0810 23:08:52.008216 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":170,"threads_started":1}
I20260810 23:08:52.012270 28108 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:52.012526 28108 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.013100 28108 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.015378 28108 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:38101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:08:52.015501 28108 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.015679 28108 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:52.016937 28108 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.017132 28108 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.017907 28108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:08:52.160909 28110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:52.161360 28110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:52.161482 28110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:52.168082 28110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:52.168187 28110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:52.168289 28110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:08:52.176999 28110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:33103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=38285
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:52.178572 28110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:52.180012 28110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:52.191793 28119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:52.191793 28118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:52.192215 28121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:52.193172 28110 server_base.cc:1034] running on GCE node
I20260810 23:08:52.193691 28110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:52.194422 28110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:52.195618 28110 hybrid_clock.cc:648] HybridClock initialized: now 1786403332195584 us; error 46 us; skew 500 ppm
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:52.199461 28110 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:08:52.200637 28110 webserver.cc:460] Webserver started at http://127.14.75.2:38285/ using document root <none> and password file <none>
I20260810 23:08:52.201370 28110 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:52.201504 28110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:52.226336 28110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:08:52.234035 28129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.235846 28110 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260810 23:08:52.236125 28110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "7870b63237bc490384102c704c7f1274"
format_stamp: "Formatted at 2026-08-10 23:08:50 on dist-test-slave-srgv"
server_key: "34ddff37abc7a44076c0771cdd395b70"
server_key_iv: "cc278e2ef08c2c7a936fa94b11245fea"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:52.236997 28110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:52.251900 28110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:52.255378 28110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:52.255606 28110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:52.256188 28110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:52.257097 28110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:52.257171 28110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.257241 28110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:52.257289 28110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.268441 28110 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33103
I20260810 23:08:52.268639 28249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33103 every 8 connection(s)
I20260810 23:08:52.269984 28110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:08:52.273988 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28110
I20260810 23:08:52.274541 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.75.20:51565/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:44383
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:38101
--webserver_port=41333
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:52.288527 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:52.272118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:55375 (local address 127.14.75.62:38101)
0810 23:08:52.272268 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:52.272273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:52.274263 (+ 1990us) server_negotiation.cc:407] Connection header received
0810 23:08:52.275553 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:52.275558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:52.275631 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:52.275687 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:52.277349 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.278232 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.280002 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.280384 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.280499 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:52.283608 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:52.283612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:52.283615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:52.283652 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:52.285666 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.286431 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.286435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.286437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.286531 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.287015 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.287019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.287020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.287268 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:52.287342 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:52.288169 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 23:08:52.288338 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:08:52.290632 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:52.272635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.2:55375)
0810 23:08:52.274071 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:52.274126 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:52.275278 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:52.276103 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:52.276112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:52.276486 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:52.277140 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:52.277158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.278417 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.278421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:52.279847 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:52.279861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.280578 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.280583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:52.280809 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:52.281579 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:52.281601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:52.283378 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:52.285842 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.285849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.285868 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.286296 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.286653 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.286658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.286660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.286889 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.287432 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:52.287439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:52.287846 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 23:08:52.289003 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0810 23:08:52.289449 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":142,"threads_started":1}
I20260810 23:08:52.292449 28250 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:52.292759 28250 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.293371 28250 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.295320 28250 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:38101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:08:52.295468 28250 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.295670 28250 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:52.296581 28250 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.296767 28250 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.297556 28250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:08:52.417208 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:52.417610 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:52.417702 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:52.422266 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:52.422364 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:52.422459 28253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:08:52.428434 28253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.75.20:51565/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:44383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=41333
--tserver_master_addrs=127.14.75.62:38101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:52.429850 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:52.431236 28253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:52.444972 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:52.445094 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:52.444972 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:52.446663 28253 server_base.cc:1034] running on GCE node
I20260810 23:08:52.447221 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:52.448187 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:52.449373 28253 hybrid_clock.cc:648] HybridClock initialized: now 1786403332449331 us; error 62 us; skew 500 ppm
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:52.453476 28253 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:08:52.454803 28253 webserver.cc:460] Webserver started at http://127.14.75.3:41333/ using document root <none> and password file <none>
I20260810 23:08:52.455621 28253 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:52.455742 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:52.477597 28253 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:08:52.480962 28271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.482494 28253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:08:52.482618 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "290539aa34cf41d5b908b6912a22831e"
format_stamp: "Formatted at 2026-08-10 23:08:51 on dist-test-slave-srgv"
server_key: "465abe7c73d3615dfaf404ebd12af664"
server_key_iv: "f1ee547db8b3b080c27e4c7aa6079516"
server_key_version: "kuduclusterkey@0"
I20260810 23:08:52.483232 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:52.500833 28253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:52.505162 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:52.505429 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:52.506203 28253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:08:52.507285 28253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:08:52.507351 28253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.507413 28253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:08:52.507443 28253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:52.520831 28253 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:44383
I20260810 23:08:52.522281 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:08:52.523181 28387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:44383 every 8 connection(s)
I20260810 23:08:52.531450 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28253
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:52.539430 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:52.524822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:43649 (local address 127.14.75.62:38101)
0810 23:08:52.524961 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:52.524965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:52.525987 (+ 1022us) server_negotiation.cc:407] Connection header received
0810 23:08:52.527013 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:52.527016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:52.527077 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:52.527158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:52.528261 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.529028 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.530571 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.530930 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.531043 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:52.534530 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:52.534535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:52.534538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:52.534569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:52.536456 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.537261 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.537265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.537266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.537368 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.537906 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.537910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.537913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.538182 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:52.538299 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:52.539099 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 23:08:52.539244 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:08:52.540663 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:52.525246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38101 (local address 127.14.75.3:43649)
0810 23:08:52.525820 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:52.525874 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:52.526784 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:52.527352 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:52.527359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:52.527632 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:52.528108 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:52.528121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.529200 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.529205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:52.530426 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:52.530437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.531228 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.531232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:52.531420 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:52.532143 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:52.532168 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:52.534308 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:52.536628 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.536639 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.536661 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.537107 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.537502 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:52.537506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:52.537508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:52.537755 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:52.538367 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:52.538375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:52.538790 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 23:08:52.539049 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 23:08:52.539457 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":222,"threads_started":1}
I20260810 23:08:52.542057 28388 heartbeater.cc:347] Connected to a master server at 127.14.75.62:38101
I20260810 23:08:52.542275 28388 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.542860 28388 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.544728 28388 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:38101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:08:52.544873 28388 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.545501 28388 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:52.546931 28388 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:52.547129 28388 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
W20260810 23:08:52.547955 28388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:08:52.549002 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:52.532360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.14.75.62:38101)
0810 23:08:52.535082 (+ 2722us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:52.535089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:52.536753 (+ 1664us) server_negotiation.cc:407] Connection header received
0810 23:08:52.536969 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:52.536976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:52.537029 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:52.537095 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:52.538229 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.538856 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.540112 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:52.540350 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:52.540460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:52.544468 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:52.544472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:52.544475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:52.544502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:52.546674 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.547474 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.547477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.547479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.547581 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:52.548092 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:52.548095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:52.548097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:52.548299 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:52.548533 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:52.548676 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:08:52.548806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2600,"thread_start_us":97,"threads_started":1}
Aug 10 23:08:52 dist-test-slave-srgv krb5kdc[26873](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:53.018153 28108 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:53.018483 28108 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:53.019647 27893 ts_manager.cc:194] Registered new tserver with Master: 1c6b8c88147749968c3e6f641479f910 (127.14.75.1:34429)
I20260810 23:08:53.021054 27893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:59743
I20260810 23:08:53.297854 28250 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:53.298214 28250 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:53.299096 27893 ts_manager.cc:194] Registered new tserver with Master: 7870b63237bc490384102c704c7f1274 (127.14.75.2:33103)
I20260810 23:08:53.299815 27893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:55375
Aug 10 23:08:53 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:53.548213 28388 heartbeater.cc:464] Registering TS with master...
I20260810 23:08:53.548579 28388 heartbeater.cc:511] Master 127.14.75.62:38101 requested a full tablet report, sending...
I20260810 23:08:53.549467 27893 ts_manager.cc:194] Registered new tserver with Master: 290539aa34cf41d5b908b6912a22831e (127.14.75.3:44383)
I20260810 23:08:53.550248 27893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:43649
I20260810 23:08:53.559998 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:08:53.572127 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.561885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.14.75.62:38101)
0810 23:08:53.562154 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.562158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.562306 (+ 148us) server_negotiation.cc:407] Connection header received
0810 23:08:53.562429 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.562432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.562482 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.562586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:53.563510 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.564357 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.565643 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.565982 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.566112 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.568014 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:53.568018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:53.568021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:53.568054 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:53.570011 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:53.570573 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:53.570578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:53.570580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:53.570685 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:53.571022 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:53.571025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:53.571027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:53.571268 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:53.571411 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.571743 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.571929 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260810 23:08:53.577090 27893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:08:53.582305 27893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:08:53.605260 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.595526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:44383 (local address 127.0.0.1:42316)
0810 23:08:53.597580 (+ 2054us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.597642 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:53.597815 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:53.599884 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:53.599887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:53.599905 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:53.600003 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.600008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.602418 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.602422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.603536 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.603548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.604729 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.604733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.604848 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.604892 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:08:53.604979 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:08:53.605097 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"mutex_wait_us":3,"thread_start_us":138,"threads_started":1}
I20260810 23:08:53.607285 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.595637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.14.75.3:44383)
0810 23:08:53.599444 (+ 3807us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.599451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.599474 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:08:53.599546 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.599551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.599732 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.600154 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:53.600366 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.602296 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.603705 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.604580 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.606745 (+ 2165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.606930 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.606988 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.607093 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3684,"thread_start_us":100,"threads_started":1}
I20260810 23:08:53.608680 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.596849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:34429 (local address 127.0.0.1:41362)
0810 23:08:53.598673 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.598688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:53.598793 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:53.599625 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:53.599632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:53.599656 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:53.599804 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.599812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.601396 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.601399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.606957 (+ 5558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.606966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.608210 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.608214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.608307 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.608333 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:08:53.608408 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:08:53.608485 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":4103,"thread_start_us":124,"threads_started":1}
I20260810 23:08:53.609228 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.597007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.14.75.1:34429)
0810 23:08:53.598523 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.598531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.598909 (+ 378us) server_negotiation.cc:407] Connection header received
0810 23:08:53.598999 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.599004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.599253 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.599418 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:53.599953 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.601181 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.607111 (+ 5930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.608046 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.608174 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.608329 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.608966 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.609085 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":98,"threads_started":1}
I20260810 23:08:53.611024 28319 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 (DEFAULT_TABLE table=test-table [id=cf158fd8513948b8beeb0cc8debe19ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:53.612265 28319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:53.612485 28038 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 (DEFAULT_TABLE table=test-table [id=cf158fd8513948b8beeb0cc8debe19ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:53.613564 28038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:53.617942 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.597625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33103 (local address 127.0.0.1:57960)
0810 23:08:53.599078 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.599093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:53.599182 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:53.603767 (+ 4585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:53.603771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:53.603792 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:53.603881 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.603887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.605327 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.605330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.606534 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.606546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.609663 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.609667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.617560 (+ 7893us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.617595 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:08:53.617685 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:08:53.617783 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":88,"threads_started":1}
I20260810 23:08:53.625178 28422 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Bootstrap starting.
I20260810 23:08:53.625317 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.598054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.14.75.2:33103)
0810 23:08:53.603033 (+ 4979us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.603041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.603067 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:08:53.603149 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.603154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.603395 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.603575 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:53.604050 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.605164 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.608407 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.609221 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.609341 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.609508 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.624972 (+ 15464us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.625124 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4853,"thread_start_us":98,"threads_started":1}
I20260810 23:08:53.628226 28422 tablet_bootstrap.cc:654] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:53.629352 28422 log.cc:824] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:53.629093 28159 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 (DEFAULT_TABLE table=test-table [id=cf158fd8513948b8beeb0cc8debe19ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:08:53.630429 28159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:53.632366 28422 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: No bootstrap required, opened a new log
I20260810 23:08:53.632584 28422 ts_tablet_manager.cc:1397] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:08:53.636803 28422 raft_consensus.cc:348] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.637112 28422 raft_consensus.cc:374] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:53.637177 28422 raft_consensus.cc:729] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290539aa34cf41d5b908b6912a22831e, State: Initialized, Role: FOLLOWER
I20260810 23:08:53.637727 28422 consensus_queue.cc:260] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.638854 28422 ts_tablet_manager.cc:1428] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 23:08:53.639230 28388 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
I20260810 23:08:53.641012 28424 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Bootstrap starting.
I20260810 23:08:53.642835 28427 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Bootstrap starting.
I20260810 23:08:53.644955 28424 tablet_bootstrap.cc:654] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:53.647271 28427 tablet_bootstrap.cc:654] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:53.647262 28424 log.cc:824] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:53.648192 28427 log.cc:824] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:53.650184 28424 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: No bootstrap required, opened a new log
I20260810 23:08:53.650452 28424 ts_tablet_manager.cc:1397] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:08:53.650605 28427 tablet_bootstrap.cc:492] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: No bootstrap required, opened a new log
I20260810 23:08:53.650799 28427 ts_tablet_manager.cc:1397] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:08:53.654743 28427 raft_consensus.cc:348] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.654748 28424 raft_consensus.cc:348] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.655062 28427 raft_consensus.cc:374] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:53.655062 28424 raft_consensus.cc:374] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:53.655123 28424 raft_consensus.cc:729] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6b8c88147749968c3e6f641479f910, State: Initialized, Role: FOLLOWER
I20260810 23:08:53.655123 28427 raft_consensus.cc:729] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7870b63237bc490384102c704c7f1274, State: Initialized, Role: FOLLOWER
I20260810 23:08:53.655725 28427 consensus_queue.cc:260] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.656957 28427 ts_tablet_manager.cc:1428] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:08:53.657373 28250 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
I20260810 23:08:53.659181 28424 consensus_queue.cc:260] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.660782 28424 ts_tablet_manager.cc:1428] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:08:53.661280 28108 heartbeater.cc:503] Master 127.14.75.62:38101 was elected leader, sending a full tablet report...
Aug 10 23:08:53 dist-test-slave-srgv krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403332, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for HTTP/127.14.75.20@KRBTEST.COM
I20260810 23:08:53.716965 28430 raft_consensus.cc:482] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:08:53.717233 28430 raft_consensus.cc:504] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.718767 28430 leader_election.cc:288] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290539aa34cf41d5b908b6912a22831e (127.14.75.3:44383), 1c6b8c88147749968c3e6f641479f910 (127.14.75.1:34429)
I20260810 23:08:53.724339 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.719494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:54325 (local address 127.14.75.3:44383)
0810 23:08:53.719625 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.719629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.719932 (+ 303us) server_negotiation.cc:407] Connection header received
0810 23:08:53.720131 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.720134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.720184 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.720230 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:53.720663 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.721473 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.723028 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.723591 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.723678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.723742 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.724123 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.724206 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:08:53.724996 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.718975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:44383 (local address 127.14.75.2:54325)
0810 23:08:53.719844 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.719858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:53.720012 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:53.720369 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:53.720371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:53.720387 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:53.720524 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.720530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.721628 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.721631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.722873 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.722886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.723842 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.723845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.723954 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.723977 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:08:53.724762 (+ 785us) client_negotiation.cc:240] Negotiation successful
0810 23:08:53.724853 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":219,"threads_started":1}
I20260810 23:08:53.725998 28339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb1f2c4064b4d569b4c9ed6b3b330d3" candidate_uuid: "7870b63237bc490384102c704c7f1274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290539aa34cf41d5b908b6912a22831e" is_pre_election: true
I20260810 23:08:53.726379 28339 raft_consensus.cc:2454] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7870b63237bc490384102c704c7f1274 in term 0.
I20260810 23:08:53.726917 28133 leader_election.cc:302] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290539aa34cf41d5b908b6912a22831e, 7870b63237bc490384102c704c7f1274; no voters:
I20260810 23:08:53.729754 28430 raft_consensus.cc:2781] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:08:53.729893 28430 raft_consensus.cc:482] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:08:53.729954 28430 raft_consensus.cc:3037] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:53.730327 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.719693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:47301 (local address 127.14.75.1:34429)
0810 23:08:53.719899 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.719906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.725859 (+ 5953us) server_negotiation.cc:407] Connection header received
0810 23:08:53.726068 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.726071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.726131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.726185 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:08:53.726744 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.727450 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.728914 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.729498 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.729576 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.729652 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.730115 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.730205 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260810 23:08:53.730876 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.718995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:34429 (local address 127.14.75.2:47301)
0810 23:08:53.725776 (+ 6781us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.725792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:53.725936 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:53.726404 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:53.726409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:53.726430 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:08:53.726590 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.726597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.727672 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.727675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.728774 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:53.728784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.729745 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.729748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:53.729830 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.729852 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:08:53.730670 (+ 818us) client_negotiation.cc:240] Negotiation successful
0810 23:08:53.730753 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6714,"thread_start_us":206,"threads_started":1}
I20260810 23:08:53.731691 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb1f2c4064b4d569b4c9ed6b3b330d3" candidate_uuid: "7870b63237bc490384102c704c7f1274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6b8c88147749968c3e6f641479f910" is_pre_election: true
I20260810 23:08:53.732030 28046 raft_consensus.cc:2454] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7870b63237bc490384102c704c7f1274 in term 0.
I20260810 23:08:53.732429 28430 raft_consensus.cc:504] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
I20260810 23:08:53.733179 28046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb1f2c4064b4d569b4c9ed6b3b330d3" candidate_uuid: "7870b63237bc490384102c704c7f1274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c6b8c88147749968c3e6f641479f910"
I20260810 23:08:53.733199 28339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb1f2c4064b4d569b4c9ed6b3b330d3" candidate_uuid: "7870b63237bc490384102c704c7f1274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290539aa34cf41d5b908b6912a22831e"
I20260810 23:08:53.733300 28046 raft_consensus.cc:3037] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:53.733343 28339 raft_consensus.cc:3037] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:53.734689 28046 raft_consensus.cc:2454] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7870b63237bc490384102c704c7f1274 in term 1.
I20260810 23:08:53.734759 28339 raft_consensus.cc:2454] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7870b63237bc490384102c704c7f1274 in term 1.
I20260810 23:08:53.735055 28132 leader_election.cc:302] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c6b8c88147749968c3e6f641479f910, 7870b63237bc490384102c704c7f1274; no voters:
I20260810 23:08:53.735472 28436 raft_consensus.cc:2781] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:08:53.735749 28436 raft_consensus.cc:686] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 1 LEADER]: Becoming Leader. State: Replica: 7870b63237bc490384102c704c7f1274, State: Running, Role: LEADER
I20260810 23:08:53.736182 28436 consensus_queue.cc:237] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } }
W20260810 23:08:53.737238 28109 tablet.cc:2369] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:08:53.740563 27893 catalog_manager.cc:5495] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7870b63237bc490384102c704c7f1274 (127.14.75.2). New cstate: current_term: 1 leader_uuid: "7870b63237bc490384102c704c7f1274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7870b63237bc490384102c704c7f1274" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33103 } health_report { overall_health: HEALTHY } } }
I20260810 23:08:53.741453 28430 leader_election.cc:288] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [CANDIDATE]: Term 1 election: Requested vote from peers 290539aa34cf41d5b908b6912a22831e (127.14.75.3:44383), 1c6b8c88147749968c3e6f641479f910 (127.14.75.1:34429)
W20260810 23:08:53.776185 28251 tablet.cc:2369] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:08:53.777014 28389 tablet.cc:2369] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:08:53.780012 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.772975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.14.75.2:33103)
0810 23:08:53.773097 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:53.773102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:53.773115 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:08:53.773226 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:53.773230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:53.773300 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:53.773442 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:08:53.773859 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.774636 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.777580 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:53.777941 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:53.778058 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:53.778562 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:08:53.779128 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:08:53.779262 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:53.779787 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 23:08:53.779860 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:08:53.787935 28339 raft_consensus.cc:1261] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 1 FOLLOWER]: Refusing update from remote peer 7870b63237bc490384102c704c7f1274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:53.787935 28046 raft_consensus.cc:1261] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 1c6b8c88147749968c3e6f641479f910 [term 1 FOLLOWER]: Refusing update from remote peer 7870b63237bc490384102c704c7f1274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:53.788740 28436 consensus_queue.cc:1035] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c6b8c88147749968c3e6f641479f910" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:53.788975 28437 consensus_queue.cc:1035] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290539aa34cf41d5b908b6912a22831e" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:53.805876 28441 mvcc.cc:205] Tried to move back new op lower bound from 7317108055184744448 to 7317108054990553088. Current Snapshot: MvccSnapshot[applied={T|T < 7317108055184744448}]
I20260810 23:08:53.814708 28444 mvcc.cc:205] Tried to move back new op lower bound from 7317108055184744448 to 7317108054990553088. Current Snapshot: MvccSnapshot[applied={T|T < 7317108055184744448}]
I20260810 23:08:53.815092 28442 mvcc.cc:205] Tried to move back new op lower bound from 7317108055184744448 to 7317108054990553088. Current Snapshot: MvccSnapshot[applied={T|T < 7317108055184744448}]
I20260810 23:08:53.823905 27893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:08:53.824136 27893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:08:53.827421 27893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efa564672e3140078c95f429f044285e: Sending DeleteTablet for 3 replicas of tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3
I20260810 23:08:53.828343 28319 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:53 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:42316
I20260810 23:08:53.829473 28038 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:53 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260810 23:08:53.830478 28159 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:08:53 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:57960
I20260810 23:08:53.832041 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27941
I20260810 23:08:53.839619 28450 tablet_replica.cc:330] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: stopping tablet replica
I20260810 23:08:53.840083 28450 raft_consensus.cc:2229] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:08:53.840518 28450 raft_consensus.cc:2258] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:08:53.843595 28450 ts_tablet_manager.cc:1905] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:08:53.846580 28452 tablet_replica.cc:330] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274: stopping tablet replica
I20260810 23:08:53.850490 28452 raft_consensus.cc:2229] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:08:53.851013 28452 raft_consensus.cc:2258] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 7870b63237bc490384102c704c7f1274 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:08:53.855049 27886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:34429 (error 108)
I20260810 23:08:53.856424 28450 ts_tablet_manager.cc:1918] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:08:53.856571 28450 log.cc:1196] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/2bb1f2c4064b4d569b4c9ed6b3b330d3
I20260810 23:08:53.856959 28450 ts_tablet_manager.cc:1939] T 2bb1f2c4064b4d569b4c9ed6b3b330d3 P 290539aa34cf41d5b908b6912a22831e: Deleting consensus metadata
I20260810 23:08:53.858518 27887 catalog_manager.cc:4841] TS 290539aa34cf41d5b908b6912a22831e (127.14.75.3:44383): tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3 (table test-table [id=cf158fd8513948b8beeb0cc8debe19ae]) successfully deleted
I20260810 23:08:53.858906 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28110
I20260810 23:08:53.869028 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28253
I20260810 23:08:53.874871 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.874251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:34429 (local address 127.0.0.1:41374)
0810 23:08:53.874674 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.874763 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
I20260810 23:08:53.875181 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:53.874508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33103 (local address 127.0.0.1:57972)
0810 23:08:53.875058 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:53.875108 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.2:33103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
W20260810 23:08:53.875562 27886 catalog_manager.cc:4584] TS 1c6b8c88147749968c3e6f641479f910 (127.14.75.1:34429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bb1f2c4064b4d569b4c9ed6b3b330d3: Network error: Client connection negotiation failed: client connection to 127.14.75.1:34429: connect: Connection refused (error 111)
I20260810 23:08:53.879695 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 27867
I20260810 23:08:53.889784 14636 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:08:54.235316 14636 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:08:54.235555 14636 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:08:54.581041 14636 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:08:54.581 UTC [26903] LOG: received smart shutdown request
2026-08-10 23:08:54.586 UTC [26903] LOG: background worker "logical replication launcher" (PID 26913) exited with exit code 1
2026-08-10 23:08:54.586 UTC [26907] LOG: shutting down
2026-08-10 23:08:54.778 UTC [26903] LOG: database system is shut down
2026-08-10T23:08:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51719 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:08:54 dist-test-slave-srgv krb5kdc[28462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:08:54 dist-test-slave-srgv krb5kdc[28462](info): set up 2 sockets
Aug 10 23:08:54 dist-test-slave-srgv krb5kdc[28462](info): commencing operation
krb5kdc: starting...
W20260810 23:08:55.982973 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:08:56.120412 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:32905
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:56.236866 28475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:56.237218 28475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:56.237301 28475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:56.241557 28475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:56.241659 28475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:56.241694 28475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:56.241734 28475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:56.241770 28475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:56.247421 28475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:32905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:56.248615 28475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:56.249701 28475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:56.258548 28483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.258548 28481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.258565 28480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:56.259097 28475 server_base.cc:1034] running on GCE node
I20260810 23:08:56.259783 28475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:56.260926 28475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:56.262130 28475 hybrid_clock.cc:648] HybridClock initialized: now 1786403336262110 us; error 35 us; skew 500 ppm
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:56.265303 28475 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:08:56.266399 28475 webserver.cc:460] Webserver started at http://127.14.75.62:37449/ using document root <none> and password file <none>
I20260810 23:08:56.267009 28475 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:56.267110 28475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:56.267356 28475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:08:56.269650 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "cbf6166ddabf4c9b9348ddb5653815d7"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "9fd9fe61e9c4628fd9b2942030430ef4"
server_key_iv: "21ca30f0e6d6fd76791e6ba2b64bc916"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.270231 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "cbf6166ddabf4c9b9348ddb5653815d7"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "9fd9fe61e9c4628fd9b2942030430ef4"
server_key_iv: "21ca30f0e6d6fd76791e6ba2b64bc916"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.274211 28475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:08:56.276764 28490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:56.278234 28475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:08:56.278396 28475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "cbf6166ddabf4c9b9348ddb5653815d7"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "9fd9fe61e9c4628fd9b2942030430ef4"
server_key_iv: "21ca30f0e6d6fd76791e6ba2b64bc916"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.278525 28475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:56.294735 28475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:56.298434 28475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:56.298655 28475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:56.306172 28475 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:32905
I20260810 23:08:56.306193 28545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:32905 every 8 connection(s)
I20260810 23:08:56.307191 28475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:08:56.308423 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28475
I20260810 23:08:56.308524 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:08:56.308821 14636 external_mini_cluster.cc:1428] Setting key b5f3d44bc3ee48a5f398be0a1a6924de
I20260810 23:08:56.310165 28546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:56.314146 28546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.61@KRBTEST.COM" created.
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:56.330188 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.316623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:32905 (local address 127.0.0.1:33544)
0810 23:08:56.317230 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.317274 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.318219 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.319038 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.319042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.319381 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.319718 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.319743 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.320933 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.320936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.321759 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.321769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.322429 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.322433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.322540 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.323198 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.323224 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.324744 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.326885 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.326896 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.326907 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.327150 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.327493 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.327496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.327498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.327618 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.328252 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.328258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.328409 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.328582 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.328998 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":162,"threads_started":1}
I20260810 23:08:56.330192 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.316623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.14.75.62:32905)
0810 23:08:56.317235 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.317253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.317393 (+ 140us) server_negotiation.cc:407] Connection header received
0810 23:08:56.318511 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.318515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.318806 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.319011 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.319889 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.320797 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.321883 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.322269 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.322442 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.324899 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.324905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.324915 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.324936 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.326762 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.327269 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.327281 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.327282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.327388 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.327727 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.327731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.327733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.328128 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.328285 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.328654 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.329045 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":484,"spinlock_wait_cycles":60160,"thread_start_us":188,"threads_started":1}
I20260810 23:08:56.332794 28546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.333811 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.333391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50720)
0810 23:08:56.333656 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.333742 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260810 23:08:56.333998 28495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
I20260810 23:08:56.334985 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.334776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50734)
0810 23:08:56.334871 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.334919 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 23:08:56.335119 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34775: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
I20260810 23:08:56.337337 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:34775
--webserver_interface=127.14.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:08:56.384413 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } attempt: 1
I20260810 23:08:56.385294 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.384984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50742)
0810 23:08:56.385140 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.385217 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:08:56.386092 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.385895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50754)
0810 23:08:56.385995 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.386042 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:08:56.386255 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34775: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
I20260810 23:08:56.454449 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } attempt: 2
I20260810 23:08:56.455369 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.455066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50766)
0810 23:08:56.455229 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.455293 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:08:56.456096 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.455884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50772)
0810 23:08:56.455987 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.456040 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:08:56.456224 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34775: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34775: connect: Connection refused (error 111)
W20260810 23:08:56.458006 28553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:56.458321 28553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:56.458425 28553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:56.462675 28553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:56.462771 28553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:56.462808 28553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:56.462864 28553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:56.462901 28553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:56.468526 28553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:56.469766 28553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:56.470825 28553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:56.479501 28561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.479553 28559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.479578 28558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:56.479887 28553 server_base.cc:1034] running on GCE node
I20260810 23:08:56.480415 28553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:56.481649 28553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:56.482865 28553 hybrid_clock.cc:648] HybridClock initialized: now 1786403336482844 us; error 29 us; skew 500 ppm
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:56.485884 28553 init.cc:377] Logged in from keytab as kudu/127.14.75.61@KRBTEST.COM (short username kudu)
I20260810 23:08:56.486907 28553 webserver.cc:460] Webserver started at http://127.14.75.61:35383/ using document root <none> and password file <none>
I20260810 23:08:56.487495 28553 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:56.487592 28553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:56.487836 28553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:08:56.490046 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data/instance:
uuid: "4c6aa64ac6dc4463b3f5980f4b51b760"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "5f0e022632fc9b42925fff964381284b"
server_key_iv: "071d253129ece9af0706bcdb83d2a713"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.490705 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal/instance:
uuid: "4c6aa64ac6dc4463b3f5980f4b51b760"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "5f0e022632fc9b42925fff964381284b"
server_key_iv: "071d253129ece9af0706bcdb83d2a713"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.494818 28553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:08:56.497524 28568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:56.498792 28553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:08:56.498945 28553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "4c6aa64ac6dc4463b3f5980f4b51b760"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "5f0e022632fc9b42925fff964381284b"
server_key_iv: "071d253129ece9af0706bcdb83d2a713"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.499076 28553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:56.521873 28553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:56.525803 28553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:56.526021 28553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:56.533679 28553 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.61:34775
I20260810 23:08:56.533701 28621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.61:34775 every 8 connection(s)
I20260810 23:08:56.534678 28553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
I20260810 23:08:56.535223 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28553
I20260810 23:08:56.535370 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-1/wal/instance
I20260810 23:08:56.535684 14636 external_mini_cluster.cc:1428] Setting key 7524280c18d6b168b875d5bc69ab0261
I20260810 23:08:56.537660 28622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:56.541585 28622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.75.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.60@KRBTEST.COM" created.
I20260810 23:08:56.547397 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } attempt: 3
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:08:56.558806 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.543611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.14.75.62:32905)
0810 23:08:56.543811 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.543817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.544726 (+ 909us) server_negotiation.cc:407] Connection header received
0810 23:08:56.545777 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.545780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.545831 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.545912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.547428 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.548287 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.549963 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.550257 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.550375 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.553451 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.553458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.553461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.553494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.555289 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.556113 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.556115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.556117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.556199 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.556579 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.556583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.556585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.556792 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.556903 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.558424 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.558647 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:08:56.559060 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.543966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:32905 (local address 127.0.0.1:33554)
0810 23:08:56.544529 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.544577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.545565 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.546077 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.546084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.546599 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.547236 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.547250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.548663 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.548666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.549841 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.549850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.550422 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.550426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.550601 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.551267 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.551293 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.553257 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.555430 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.555446 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.555461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.555812 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.556312 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.556316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.556318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.556476 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.556904 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.556913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.557234 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.557507 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.557943 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":119,"threads_started":1}
I20260810 23:08:56.559322 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.548090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.14.75.61:34775)
0810 23:08:56.548586 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.548600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.548638 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:08:56.548689 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.548696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.548918 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.549099 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.549602 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.550851 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.551781 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.552160 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.552295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.554813 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.554822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.554841 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.554875 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.557080 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.557630 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.557638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.557641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.557736 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.558080 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.558084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.558086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.558464 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.558603 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.558991 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.559145 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":181,"threads_started":1}
I20260810 23:08:56.559134 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.547896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50788)
0810 23:08:56.548497 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.548509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.548564 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.549112 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.549115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.549328 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.549474 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.549482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.550980 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.550985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.551654 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.551664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.552316 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.552323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.552464 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.552994 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.553015 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.554631 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.557195 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.557198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.557200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.557486 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.557853 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.557856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.557858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.557954 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.558583 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.558586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.558745 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.558852 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.558982 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515}
I20260810 23:08:56.560644 28622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.561628 28546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.562350 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.562110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50566)
0810 23:08:56.562221 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.562278 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:08:56.563114 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.562837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50574)
0810 23:08:56.562971 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.563022 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 23:08:56.563297 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
I20260810 23:08:56.565033 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:46483
--webserver_interface=127.14.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:08:56.569379 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.561176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50796)
0810 23:08:56.561273 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.561288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.561369 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.561680 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.561683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.561882 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.561966 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.561971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.562877 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.562880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.563591 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.563605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.564064 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.564067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.564190 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.564712 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.564727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.566102 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.567886 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.567889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.567891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.568084 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.568355 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.568360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.568362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.568446 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.568828 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.568831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.568917 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.569026 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.569189 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":5}
I20260810 23:08:56.569379 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.561255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.14.75.61:34775)
0810 23:08:56.561496 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.561501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.561514 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:08:56.561552 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.561555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.561597 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.561675 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.562079 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.562763 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.563702 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.563939 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.564066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.566241 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.566244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.566247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.566269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.567763 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.568180 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.568186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.568188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.568265 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.568544 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.568546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.568548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.568733 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.568839 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.569094 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.569242 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":167}
I20260810 23:08:56.570313 28622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.571100 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.570821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50580)
0810 23:08:56.570947 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.571037 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:08:56.571774 28569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
I20260810 23:08:56.572678 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.572464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50586)
0810 23:08:56.572572 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.572624 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:08:56.572876 28622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
I20260810 23:08:56.594481 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 1
I20260810 23:08:56.595448 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.595085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50592)
0810 23:08:56.595274 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.595351 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:08:56.596238 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.595990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50598)
0810 23:08:56.596111 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.596168 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:08:56.596405 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
I20260810 23:08:56.622226 28622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 1
I20260810 23:08:56.623268 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.622900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50608)
0810 23:08:56.623105 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.623174 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:08:56.624034 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.623804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50624)
0810 23:08:56.623913 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.623969 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:08:56.624174 28622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
I20260810 23:08:56.671630 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 2
I20260810 23:08:56.672580 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.672237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50632)
0810 23:08:56.672431 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.672499 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 23:08:56.673377 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.673150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50638)
0810 23:08:56.673262 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.673317 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:08:56.673537 28546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
W20260810 23:08:56.685822 28629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:56.686153 28629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:56.686264 28629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:56.690514 28629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:56.690606 28629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:56.690640 28629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:56.690698 28629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:56.690734 28629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:56.692337 28622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 2
I20260810 23:08:56.693142 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.692844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50654)
0810 23:08:56.692988 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.693058 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:08:56.693910 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.693692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50656)
0810 23:08:56.693800 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.693854 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:08:56.694034 28622 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46483: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46483: connect: Connection refused (error 111)
I20260810 23:08:56.696631 28629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:32905,127.14.75.61:34775,127.14.75.60:46483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:56.697904 28629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:56.699075 28629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:56.708070 28638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.708074 28636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:56.708070 28635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:56.708340 28629 server_base.cc:1034] running on GCE node
I20260810 23:08:56.709120 28629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:56.710391 28629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:56.711565 28629 hybrid_clock.cc:648] HybridClock initialized: now 1786403336711528 us; error 54 us; skew 500 ppm
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:56.714656 28629 init.cc:377] Logged in from keytab as kudu/127.14.75.60@KRBTEST.COM (short username kudu)
I20260810 23:08:56.715664 28629 webserver.cc:460] Webserver started at http://127.14.75.60:46099/ using document root <none> and password file <none>
I20260810 23:08:56.716272 28629 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:56.716341 28629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:56.716583 28629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:08:56.718734 28629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data/instance:
uuid: "6c2a300c704743739708ca1a388519ad"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "4d77a85c87ac237057929ca258f6085a"
server_key_iv: "a6953cded5fe564baa685cfa509a7a5f"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.719329 28629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal/instance:
uuid: "6c2a300c704743739708ca1a388519ad"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "4d77a85c87ac237057929ca258f6085a"
server_key_iv: "a6953cded5fe564baa685cfa509a7a5f"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.723217 28629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:08:56.725703 28645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:56.726866 28629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:08:56.727018 28629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "6c2a300c704743739708ca1a388519ad"
format_stamp: "Formatted at 2026-08-10 23:08:56 on dist-test-slave-srgv"
server_key: "4d77a85c87ac237057929ca258f6085a"
server_key_iv: "a6953cded5fe564baa685cfa509a7a5f"
server_key_version: "encryptionkey@0"
I20260810 23:08:56.727150 28629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:56.742169 28629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:56.745517 28629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:56.745740 28629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:56.753453 28629 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.60:46483
I20260810 23:08:56.753530 28698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.60:46483 every 8 connection(s)
I20260810 23:08:56.754452 28629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
I20260810 23:08:56.757232 28699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:56.761090 28699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.762452 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28629
I20260810 23:08:56.762559 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786403171263156-14636-0/minicluster-data/master-2/wal/instance
I20260810 23:08:56.762857 14636 external_mini_cluster.cc:1428] Setting key 675d8276ad86095a7db8b68872dc2270
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:08:56.772701 28546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 3
I20260810 23:08:56.776721 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.763410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.14.75.62:32905)
0810 23:08:56.763568 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.763573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.764518 (+ 945us) server_negotiation.cc:407] Connection header received
0810 23:08:56.765479 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.765486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.765549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.765623 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.766715 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.767482 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.768906 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.769249 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.769356 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.772441 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.772448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.772454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.772492 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.774333 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.775012 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.775017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.775019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.775121 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.775489 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.775496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.775498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.775784 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.775892 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.776497 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.776596 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:08:56.778128 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.763851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:32905 (local address 127.0.0.1:33560)
0810 23:08:56.764356 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.764406 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.765231 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.765714 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.765723 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.766191 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.766569 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.766582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.767609 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.767612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.768780 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.768790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.769397 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.769400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.769619 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.770413 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.770434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.772249 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.774452 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.774463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.774474 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.774887 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.775230 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.775233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.775235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.775385 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.775889 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.775896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.776254 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.776463 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.776809 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":22,"thread_start_us":173,"threads_started":1}
I20260810 23:08:56.778650 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.763851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.14.75.60:46483)
0810 23:08:56.764389 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.764404 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.764434 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:08:56.765166 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.765182 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.765398 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.765607 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.766530 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.767748 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.768905 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.769521 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.769613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.773004 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.773015 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.773030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.773060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.775078 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.775569 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.775575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.775577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.775661 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.776843 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.776846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.776848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.777107 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.777324 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.777968 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.778113 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":441,"spinlock_wait_cycles":12416,"thread_start_us":189,"threads_started":1}
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:08:56.779711 28699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.784009 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.773213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50672)
0810 23:08:56.773490 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.773510 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.773592 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.774622 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.774625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.774800 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.774925 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.774934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.775958 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.775961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.776705 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.776716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.777452 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.777455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.777542 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.778021 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.778036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.779571 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.782387 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.782391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.782393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.782665 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.782951 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.782954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.782956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.783048 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.783500 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.783503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.783603 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.783715 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.783846 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260810 23:08:56.784065 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.773296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.14.75.60:46483)
0810 23:08:56.774369 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.774381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.774396 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:08:56.774429 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.774432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.774486 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.774584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.775056 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.775827 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.776969 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.777279 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.777453 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.779697 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.779701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.779704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.779728 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.782149 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.782774 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.782778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.782780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.782869 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.783155 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.783161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.783164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.783402 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.783497 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.783793 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.783925 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":130,"threads_started":1}
I20260810 23:08:56.785202 28622 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } attempt: 3
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:56.787904 28546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: Bootstrap starting.
I20260810 23:08:56.790740 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.780193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34775 (local address 127.0.0.1:50810)
0810 23:08:56.780383 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.780396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.780462 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.780781 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.780784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.781254 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.781397 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.781431 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.782230 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.782236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.783079 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.783090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.784280 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.784286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.784384 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.785108 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.785118 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.786701 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.788736 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.788739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.788741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.788979 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.789328 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.789332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.789334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.789457 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.789985 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.789987 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.790084 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.790426 (+ 342us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.790569 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 23:08:56.790839 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.780318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.14.75.61:34775)
0810 23:08:56.780444 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.780447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.780460 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:08:56.780577 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.780582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.780653 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.780748 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.781536 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.782133 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.783289 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.783643 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.783769 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.786842 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.786846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.786849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.786875 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.788546 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.789098 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.789101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.789104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.789197 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.789577 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.789583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.789587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.789835 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.789909 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.790284 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.790680 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:08:56.791152 28546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:56.791771 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.781149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.14.75.62:32905)
0810 23:08:56.781259 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.781263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.781333 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:08:56.781463 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.781468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.781521 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.782420 (+ 899us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.782496 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.783121 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.784349 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.784714 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.784828 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.787913 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.787916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.787918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.787939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.789456 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.789924 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.789928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.789930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.790018 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.790334 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.790338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.790340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.790556 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.790767 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.791498 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.791616 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:08:56.792203 28699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:08:56.792124 28546 log.cc:824] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:08:56.795977 28546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: No bootstrap required, opened a new log
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
Aug 10 23:08:56 dist-test-slave-srgv krb5kdc[28462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:08:56.800140 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.785649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50678)
0810 23:08:56.785896 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.785909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.785985 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.790269 (+ 4284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.790275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.790481 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.790582 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.790592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.792016 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.792019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.792776 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.792786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.793472 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.793478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.793568 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.794093 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.794109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.795684 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.798396 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.798399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.798402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.798665 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.798984 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.798990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.798992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.799078 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.799569 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.799572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.799681 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.799767 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.799919 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260810 23:08:56.800745 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.789558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.14.75.60:46483)
0810 23:08:56.790008 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.790013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.790025 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:08:56.790063 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.790068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.790117 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.791063 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.791172 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.791876 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.792933 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.793277 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.793488 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.795832 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.795836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.795838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.795862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.798258 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.798794 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.798797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.798798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.798883 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.799208 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.799211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.799213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.799449 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.800335 (+ 886us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.800415 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.800548 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"spinlock_wait_cycles":1792}
I20260810 23:08:56.803244 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.792821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.14.75.60:46483)
0810 23:08:56.792948 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.792952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.793627 (+ 675us) server_negotiation.cc:407] Connection header received
0810 23:08:56.793794 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.793797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.793841 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.793891 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.794422 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.795192 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.796360 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.796694 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.796796 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.799602 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.799608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.799611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.799640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.801453 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.801932 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.801935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.801937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.802011 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.802287 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.802292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.802294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.802516 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.802628 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.802923 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.803046 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:08:56.803699 28702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.792687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46483 (local address 127.0.0.1:50690)
0810 23:08:56.793570 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0810 23:08:56.793583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:08:56.793697 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:08:56.794016 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:08:56.794019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:08:56.794206 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:08:56.794306 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.794313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.795317 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.795320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.796254 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:08:56.796263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.796814 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.796819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:08:56.796925 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.797367 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:08:56.797381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:08:56.799450 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:08:56.801574 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.801578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.801580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.801826 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.802109 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:08:56.802111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:08:56.802113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:08:56.802192 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:08:56.802636 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:08:56.802641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:08:56.802742 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:08:56.803378 (+ 636us) client_negotiation.cc:240] Negotiation successful
0810 23:08:56.803508 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812}
I20260810 23:08:56.803938 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.793066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.14.75.61:34775)
0810 23:08:56.793194 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.793198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.793324 (+ 126us) server_negotiation.cc:407] Connection header received
0810 23:08:56.793499 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.793502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.793553 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.793638 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.794450 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.795121 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.796227 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.796518 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.796640 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.799693 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.799697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.799699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.799736 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.801970 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.802470 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.802474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.802476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.802580 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.802887 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.802892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.802894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.803093 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.803339 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.803618 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.803727 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":1408}
I20260810 23:08:56.804294 28622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: Bootstrap starting.
I20260810 23:08:56.807204 28699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: Bootstrap starting.
I20260810 23:08:56.807277 28622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:56.808007 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:08:56.808164 28622 log.cc:824] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:56.809895 28699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: Neither blocks nor log segments found. Creating new log.
I20260810 23:08:56.809918 28546 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.810195 28546 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:56.810254 28546 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf6166ddabf4c9b9348ddb5653815d7, State: Initialized, Role: FOLLOWER
I20260810 23:08:56.810782 28699 log.cc:824] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:56.810786 28546 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.811065 28622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: No bootstrap required, opened a new log
I20260810 23:08:56.812114 28546 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:08:56.814553 28712 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:56.814723 28712 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:56.815665 28622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.815902 28622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:56.815963 28622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6aa64ac6dc4463b3f5980f4b51b760, State: Initialized, Role: FOLLOWER
I20260810 23:08:56.816448 28622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.816860 28699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: No bootstrap required, opened a new log
I20260810 23:08:56.818046 28622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:08:56.821367 28699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.821696 28699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:56.821765 28699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2a300c704743739708ca1a388519ad, State: Initialized, Role: FOLLOWER
I20260810 23:08:56.822258 28699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:56.822512 28703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.810468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50694 (local address 127.14.75.60:46483)
0810 23:08:56.810590 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.810595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.811170 (+ 575us) server_negotiation.cc:407] Connection header received
0810 23:08:56.811381 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.811384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.811433 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.811527 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.812455 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.813246 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.814606 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.814965 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.815072 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.818136 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.818140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.818143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.818171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.820070 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.820617 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.820623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.820625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.820741 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.821069 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.821072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.821075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.821297 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.821443 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.822197 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.822335 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:08:56.823700 28699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:08:56.824064 28741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:08:56.824157 28741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:08:56.824429 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:56.824544 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:08:56.827118 28728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:08:56.827234 28728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:08:56.827621 28743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:56.827777 28743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:56.818176 28475 master_runner.cc:186] Error getting master registration for 127.14.75.61:34775: OK, instance_id { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" instance_seqno: 1786403336531517 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:08:56.828483 28475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:08:56.832010 28553 master_runner.cc:263] No leader master found from master 4c6aa64ac6dc4463b3f5980f4b51b760
I20260810 23:08:56.832086 28553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:08:56.833655 28629 master_runner.cc:263] No leader master found from master 6c2a300c704743739708ca1a388519ad
I20260810 23:08:56.833735 28629 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:08:56.838335 28755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:08:56.838436 28755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:08:56.839270 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.815478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.14.75.61:34775)
0810 23:08:56.815617 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.815621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.819292 (+ 3671us) server_negotiation.cc:407] Connection header received
0810 23:08:56.823726 (+ 4434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.823729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.823781 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.823860 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.829883 (+ 6023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.830616 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.831813 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.832170 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.832249 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.834417 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.834422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.834427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.834460 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.837183 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.837726 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.837730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.837732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.837828 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.838154 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.838158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.838160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.838388 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.838627 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.838967 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.839086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:08:56.840001 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:56.815232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.14.75.62:32905)
0810 23:08:56.815387 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:56.815392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:56.817051 (+ 1659us) server_negotiation.cc:407] Connection header received
0810 23:08:56.818752 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:56.818755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:56.818807 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:56.818883 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:56.831129 (+ 12246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.831859 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.833530 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:56.833911 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:56.834026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:56.835876 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:56.835880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:56.835883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:56.835914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:56.838056 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.838560 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.838564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.838568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.838677 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:56.838999 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:56.839002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:56.839004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:56.839222 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:56.839342 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:56.839688 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 23:08:56.839860 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 23:08:56.997164 28651 tablet.cc:2369] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:08:57.027726 28574 tablet.cc:2369] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:08:57.050503 28498 tablet.cc:2369] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:08:57.107292 28743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:08:57.107517 28743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:57.108500 28743 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbf6166ddabf4c9b9348ddb5653815d7 (127.14.75.62:32905), 4c6aa64ac6dc4463b3f5980f4b51b760 (127.14.75.61:34775)
I20260810 23:08:57.109089 28596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c2a300c704743739708ca1a388519ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" is_pre_election: true
I20260810 23:08:57.109078 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c2a300c704743739708ca1a388519ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" is_pre_election: true
I20260810 23:08:57.109411 28596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2a300c704743739708ca1a388519ad in term 0.
I20260810 23:08:57.109496 28520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2a300c704743739708ca1a388519ad in term 0.
I20260810 23:08:57.109941 28648 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6aa64ac6dc4463b3f5980f4b51b760, 6c2a300c704743739708ca1a388519ad; no voters:
I20260810 23:08:57.110229 28743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:08:57.110322 28743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:08:57.110369 28743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:57.111434 28743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:57.111830 28743 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [CANDIDATE]: Term 1 election: Requested vote from peers cbf6166ddabf4c9b9348ddb5653815d7 (127.14.75.62:32905), 4c6aa64ac6dc4463b3f5980f4b51b760 (127.14.75.61:34775)
I20260810 23:08:57.112202 28520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c2a300c704743739708ca1a388519ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf6166ddabf4c9b9348ddb5653815d7"
I20260810 23:08:57.112205 28596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c2a300c704743739708ca1a388519ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760"
I20260810 23:08:57.112339 28520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:57.112339 28596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:57.113735 28520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2a300c704743739708ca1a388519ad in term 1.
I20260810 23:08:57.113726 28596 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2a300c704743739708ca1a388519ad in term 1.
I20260810 23:08:57.114085 28648 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6aa64ac6dc4463b3f5980f4b51b760, 6c2a300c704743739708ca1a388519ad; no voters:
I20260810 23:08:57.114338 28743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:08:57.114594 28743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [term 1 LEADER]: Becoming Leader. State: Replica: 6c2a300c704743739708ca1a388519ad, State: Running, Role: LEADER
I20260810 23:08:57.114934 28743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } }
I20260810 23:08:57.116911 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2a300c704743739708ca1a388519ad. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.117053 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:57.117511 28759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:08:57.118875 28759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:08:57.123948 28520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [term 1 FOLLOWER]: Refusing update from remote peer 6c2a300c704743739708ca1a388519ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:57.123962 28596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [term 1 FOLLOWER]: Refusing update from remote peer 6c2a300c704743739708ca1a388519ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:08:57.124785 28743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:57.124996 28757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:08:57.129122 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2a300c704743739708ca1a388519ad. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.129274 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:57.131407 28743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.131541 28743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:57.132073 28729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.132194 28729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:57.132740 28759 catalog_manager.cc:1347] Generated new cluster ID: 3490a1aad9794f769cc792d87cce82f7
I20260810 23:08:57.132812 28759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:08:57.132886 28761 mvcc.cc:205] Tried to move back new op lower bound from 7317108068851843072 to 7317108068825296896. Current Snapshot: MvccSnapshot[applied={T|T < 7317108068851843072}]
I20260810 23:08:57.133782 28712 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2a300c704743739708ca1a388519ad. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.133927 28712 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:57.135249 28757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.135365 28757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:57.136400 28712 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2a300c704743739708ca1a388519ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf6166ddabf4c9b9348ddb5653815d7" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32905 } } peers { permanent_uuid: "4c6aa64ac6dc4463b3f5980f4b51b760" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34775 } } peers { permanent_uuid: "6c2a300c704743739708ca1a388519ad" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46483 } } }
I20260810 23:08:57.136520 28712 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:08:57.154593 28759 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:08:57.155812 28759 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:08:57.165601 28759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2a300c704743739708ca1a388519ad: Generated new TSK 0
I20260810 23:08:57.166632 28759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:08:57.825659 28741 catalog_manager.cc:1259] Loaded cluster ID: 3490a1aad9794f769cc792d87cce82f7
I20260810 23:08:57.825750 28741 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: loading cluster ID for follower catalog manager: success
I20260810 23:08:57.827800 28741 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: acquiring CA information for follower catalog manager: success
I20260810 23:08:57.828694 28728 catalog_manager.cc:1259] Loaded cluster ID: 3490a1aad9794f769cc792d87cce82f7
I20260810 23:08:57.828773 28728 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: loading cluster ID for follower catalog manager: success
I20260810 23:08:57.829366 28741 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c6aa64ac6dc4463b3f5980f4b51b760: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:08:57.830636 28728 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: acquiring CA information for follower catalog manager: success
I20260810 23:08:57.831914 28728 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbf6166ddabf4c9b9348ddb5653815d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:08:58.229791 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28475
I20260810 23:08:58.237635 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28553
I20260810 23:08:58.244598 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28629
2026-08-10T23:08:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3338 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:08:58 dist-test-slave-srgv krb5kdc[28774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:08:58 dist-test-slave-srgv krb5kdc[28774](info): set up 2 sockets
Aug 10 23:08:58 dist-test-slave-srgv krb5kdc[28774](info): commencing operation
krb5kdc: starting...
W20260810 23:08:59.322587 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:08:59.461607 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:37989
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:08:59.580343 28787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:08:59.580646 28787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:08:59.580688 28787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:08:59.584975 28787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:08:59.585048 28787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:08:59.585080 28787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:08:59.585099 28787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:08:59.585115 28787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:08:59.590853 28787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:08:59.592034 28787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:08:59.593143 28787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:08:59.601751 28794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:59.601922 28787 server_base.cc:1034] running on GCE node
W20260810 23:08:59.601830 28793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:08:59.601771 28796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:08:59.602883 28787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:08:59.604138 28787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:08:59.605348 28787 hybrid_clock.cc:648] HybridClock initialized: now 1786403339605320 us; error 43 us; skew 500 ppm
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403339, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:08:59.608271 28787 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:08:59.609272 28787 webserver.cc:460] Webserver started at http://127.14.75.62:38553/ using document root <none> and password file <none>
I20260810 23:08:59.610009 28787 fs_manager.cc:359] Metadata directory not provided
I20260810 23:08:59.610083 28787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:08:59.610392 28787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:08:59.612650 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "9797bd546faf453c94d1577fee53ae4d"
format_stamp: "Formatted at 2026-08-10 23:08:59 on dist-test-slave-srgv"
server_key: "0b16ab49aee066dff246960c8325c983"
server_key_iv: "8e4efca802d2b851013349c0b8aa5fa4"
server_key_version: "encryptionkey@0"
I20260810 23:08:59.613197 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "9797bd546faf453c94d1577fee53ae4d"
format_stamp: "Formatted at 2026-08-10 23:08:59 on dist-test-slave-srgv"
server_key: "0b16ab49aee066dff246960c8325c983"
server_key_iv: "8e4efca802d2b851013349c0b8aa5fa4"
server_key_version: "encryptionkey@0"
I20260810 23:08:59.617144 28787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:08:59.619863 28803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:08:59.621213 28787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:08:59.621368 28787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "9797bd546faf453c94d1577fee53ae4d"
format_stamp: "Formatted at 2026-08-10 23:08:59 on dist-test-slave-srgv"
server_key: "0b16ab49aee066dff246960c8325c983"
server_key_iv: "8e4efca802d2b851013349c0b8aa5fa4"
server_key_version: "encryptionkey@0"
I20260810 23:08:59.621531 28787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:08:59.654677 28787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:08:59.658054 28787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:08:59.658226 28787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:08:59.665503 28787 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:37989
I20260810 23:08:59.665537 28860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:37989 every 8 connection(s)
I20260810 23:08:59.666517 28787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:08:59.669516 28861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:08:59.669624 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28787
I20260810 23:08:59.669759 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:08:59.670066 14636 external_mini_cluster.cc:1428] Setting key 213c816384ca4cf5d86cbc26a90fe3a9
I20260810 23:08:59.677049 28861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d: Bootstrap starting.
I20260810 23:08:59.679556 28861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d: Neither blocks nor log segments found. Creating new log.
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:08:59.680369 28861 log.cc:824] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:08:59.683175 28861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d: No bootstrap required, opened a new log
I20260810 23:08:59.686393 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:08:59.671557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.14.75.62:37989)
0810 23:08:59.672033 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:08:59.672054 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:08:59.672098 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:08:59.672998 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:08:59.673023 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:08:59.673504 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:08:59.673890 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:08:59.674623 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:59.676145 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:59.677047 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:08:59.677748 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:08:59.677909 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:08:59.680559 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:08:59.680565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:08:59.680583 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:08:59.680616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:08:59.682927 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:59.683459 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:59.683465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:59.683471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:59.683633 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:08:59.683933 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:08:59.683936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:08:59.683938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:08:59.684483 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:08:59.684665 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:08:59.684934 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 23:08:59.685266 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260810 23:08:59.687867 28861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } }
I20260810 23:08:59.688118 28861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:08:59.688179 28861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9797bd546faf453c94d1577fee53ae4d, State: Initialized, Role: FOLLOWER
I20260810 23:08:59.688635 28861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } }
I20260810 23:08:59.688832 28861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:08:59.688900 28861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:08:59.688999 28861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:08:59.689468 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:08:59.690253 28861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } }
I20260810 23:08:59.690626 28861 leader_election.cc:302] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9797bd546faf453c94d1577fee53ae4d; no voters:
I20260810 23:08:59.690989 28861 leader_election.cc:288] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:08:59.691118 28867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:08:59.691382 28867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [term 1 LEADER]: Becoming Leader. State: Replica: 9797bd546faf453c94d1577fee53ae4d, State: Running, Role: LEADER
I20260810 23:08:59.691706 28867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } }
I20260810 23:08:59.692255 28861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:08:59.693470 28869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9797bd546faf453c94d1577fee53ae4d. Latest consensus state: current_term: 1 leader_uuid: "9797bd546faf453c94d1577fee53ae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } } }
I20260810 23:08:59.693470 28868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9797bd546faf453c94d1577fee53ae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9797bd546faf453c94d1577fee53ae4d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 37989 } } }
I20260810 23:08:59.693653 28869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:59.693653 28868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d [sys.catalog]: This master's current role is: LEADER
I20260810 23:08:59.694103 28876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:08:59.696810 28876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:08:59.702977 28876 catalog_manager.cc:1347] Generated new cluster ID: c4c320df73d34dddb8eaf61823ad6cb7
I20260810 23:08:59.703068 28876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:08:59.708452 28876 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:08:59.709393 28876 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:08:59.714980 28876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9797bd546faf453c94d1577fee53ae4d: Generated new TSK 0
I20260810 23:08:59.715704 28876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:08:59.746179 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28787
2026-08-10T23:08:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1499 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28891](info): set up 2 sockets
Aug 10 23:08:59 dist-test-slave-srgv krb5kdc[28891](info): commencing operation
krb5kdc: starting...
W20260810 23:09:00.814603 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:09:00 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:09:00.946573 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:01.064807 28905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:01.065114 28905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:01.065157 28905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:01.069375 28905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:09:01.069489 28905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:01.069510 28905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:01.069528 28905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:09:01.069545 28905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:09:01.075263 28905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:01.076470 28905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:01.077585 28905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:01.086898 28913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.086840 28911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.086869 28910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:01.087021 28905 server_base.cc:1034] running on GCE node
I20260810 23:09:01.087894 28905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:01.089102 28905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:01.090281 28905 hybrid_clock.cc:648] HybridClock initialized: now 1786403341090262 us; error 32 us; skew 500 ppm
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:01.096997 28905 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:09:01.098116 28905 webserver.cc:460] Webserver started at http://127.14.75.62:34553/ using document root <none> and password file <none>
I20260810 23:09:01.098727 28905 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:01.098798 28905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:01.099195 28905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:01.101619 28905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "ffda23007075428c90158bfe73ffeccf"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "c7ba2352c78ebf773f9945198840aa03"
server_key_iv: "3ba98025597fa4cf62e152486c1cfc34"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.102259 28905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "ffda23007075428c90158bfe73ffeccf"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "c7ba2352c78ebf773f9945198840aa03"
server_key_iv: "3ba98025597fa4cf62e152486c1cfc34"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.106559 28905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:09:01.109275 28920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.110633 28905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:09:01.110803 28905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "ffda23007075428c90158bfe73ffeccf"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "c7ba2352c78ebf773f9945198840aa03"
server_key_iv: "3ba98025597fa4cf62e152486c1cfc34"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.110945 28905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:01.135921 28905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:01.139583 28905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:01.139827 28905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:01.147248 28905 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42755
I20260810 23:09:01.147571 28974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42755 every 8 connection(s)
I20260810 23:09:01.148303 28905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:09:01.151176 28975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:01.154747 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 28905
I20260810 23:09:01.154897 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:09:01.155200 14636 external_mini_cluster.cc:1428] Setting key ed900978eda4955d15b36f33a26a8029
I20260810 23:09:01.157588 28975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap starting.
I20260810 23:09:01.160126 28975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:01.160918 28975 log.cc:824] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:01.162989 28975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: No bootstrap required, opened a new log
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:01.167089 28975 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:01.167338 28975 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:01.167374 28975 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Initialized, Role: FOLLOWER
I20260810 23:09:01.167840 28975 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:01.167997 28975 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:09:01.168090 28975 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:09:01.168224 28975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:01.169653 28975 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:01.170045 28975 leader_election.cc:302] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffda23007075428c90158bfe73ffeccf; no voters:
I20260810 23:09:01.170423 28975 leader_election.cc:288] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:09:01.170567 28980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:01.170806 28980 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 LEADER]: Becoming Leader. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Running, Role: LEADER
I20260810 23:09:01.171181 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.156754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.14.75.62:42755)
0810 23:09:01.157288 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.157304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.157344 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:09:01.158175 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.158195 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.158551 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.158914 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:01.159808 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.160894 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.161875 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.162395 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.162578 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.165501 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:01.165507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:01.165522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:01.165554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:01.167645 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.168164 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.168171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.168177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.168342 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.168640 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.168646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.168650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.169192 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:01.169394 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.169781 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.170113 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":157,"threads_started":1}
I20260810 23:09:01.171175 28980 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:01.171769 28975 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:09:01.172920 28981 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:01.173054 28981 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:01.173393 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffda23007075428c90158bfe73ffeccf. Latest consensus state: current_term: 1 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:01.173527 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:01.173817 28989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:09:01.176535 28989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:09:01.182328 28989 catalog_manager.cc:1347] Generated new cluster ID: 362d3d85e4f74ab1af0d443de73780ef
I20260810 23:09:01.182417 28989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:09:01.194633 28989 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:09:01.195981 28989 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:09:01.202837 28989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Generated new TSK 0
I20260810 23:09:01.203804 28989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:09:01.237022 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:01.356307 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:01.356606 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:01.356654 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:01.361001 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:01.361089 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:01.361155 29002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:09:01.367126 29002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:01.368306 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:01.369364 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:09:01.379781 29002 server_base.cc:1034] running on GCE node
W20260810 23:09:01.379729 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.379743 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.379729 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:01.380509 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:01.381297 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:01.382508 29002 hybrid_clock.cc:648] HybridClock initialized: now 1786403341382494 us; error 36 us; skew 500 ppm
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:01.385555 29002 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:09:01.386474 29002 webserver.cc:460] Webserver started at http://127.14.75.1:36493/ using document root <none> and password file <none>
I20260810 23:09:01.387075 29002 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:01.387171 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:01.387408 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:01.389686 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "365b6146a7a6479f83acb9718a1a3879"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "4a6af9286d415e2acf3456c41dc60ce1"
server_key_iv: "5368dba2f77fd53dd657ce4cbb624675"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.390278 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "365b6146a7a6479f83acb9718a1a3879"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "4a6af9286d415e2acf3456c41dc60ce1"
server_key_iv: "5368dba2f77fd53dd657ce4cbb624675"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.394191 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:09:01.396658 29017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.398002 29002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:09:01.398159 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "365b6146a7a6479f83acb9718a1a3879"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "4a6af9286d415e2acf3456c41dc60ce1"
server_key_iv: "5368dba2f77fd53dd657ce4cbb624675"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.398283 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:01.428398 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:01.432154 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:01.432391 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:01.433069 29002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:01.433976 29002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:01.434058 29002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.434134 29002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:01.434181 29002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.443789 29002 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:33455
I20260810 23:09:01.443812 29134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:33455 every 8 connection(s)
I20260810 23:09:01.444777 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:09:01.445159 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29002
I20260810 23:09:01.445323 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:09:01.445708 14636 external_mini_cluster.cc:1428] Setting key 6040d302476b7400e51e7cee37ec26cb
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:01.466292 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.447469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:47301 (local address 127.14.75.62:42755)
0810 23:09:01.447645 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.447650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.449660 (+ 2010us) server_negotiation.cc:407] Connection header received
0810 23:09:01.451641 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.451644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.451692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.451761 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:01.453609 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.454151 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.455982 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.456229 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.456316 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.461655 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:01.461660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:01.461662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:01.461700 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:01.463541 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.464171 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.464178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.464182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.464288 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.464647 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.464653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.464658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.464974 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:01.465091 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.465865 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.466034 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:09:01.467080 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.447242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:47301)
0810 23:09:01.449494 (+ 2252us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.449556 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.451382 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.452011 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.452023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.452623 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:01.453394 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.453454 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.454296 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.454299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.455833 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.455849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.456418 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.456425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.456679 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.457398 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:01.457444 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:01.461377 (+ 3933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:01.463700 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.463706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.463721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.464020 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.464387 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.464390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.464391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.464535 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.465086 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:01.465092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:01.465532 (+ 440us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.465711 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.466028 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"thread_start_us":174,"threads_started":1}
I20260810 23:09:01.469933 29135 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:01.470252 29135 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:01.470796 29135 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:01.472460 28937 ts_manager.cc:194] Registered new tserver with Master: 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455)
I20260810 23:09:01.473875 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:47301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:09:01.481287 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:01.598407 29140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:01.598706 29140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:01.598752 29140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:01.603367 29140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:01.603452 29140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:01.603519 29140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:09:01.609578 29140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:01.610718 29140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:01.611726 29140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:01.621865 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.621847 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.621852 29148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:01.622362 29140 server_base.cc:1034] running on GCE node
I20260810 23:09:01.622821 29140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:01.623503 29140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:01.624689 29140 hybrid_clock.cc:648] HybridClock initialized: now 1786403341624653 us; error 48 us; skew 500 ppm
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:01.627774 29140 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:09:01.628767 29140 webserver.cc:460] Webserver started at http://127.14.75.2:34299/ using document root <none> and password file <none>
I20260810 23:09:01.629314 29140 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:01.629375 29140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:01.629632 29140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:01.631703 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "983e2e7717d24f87bdfb37a18661757e"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "f88cd05066f71a39754095614b348e28"
server_key_iv: "64d097c66da00935bb7fd8045cd73458"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.632237 29140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "983e2e7717d24f87bdfb37a18661757e"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "f88cd05066f71a39754095614b348e28"
server_key_iv: "64d097c66da00935bb7fd8045cd73458"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.636086 29140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 23:09:01.638852 29155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.640220 29140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:09:01.640380 29140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "983e2e7717d24f87bdfb37a18661757e"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "f88cd05066f71a39754095614b348e28"
server_key_iv: "64d097c66da00935bb7fd8045cd73458"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.640511 29140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:01.658244 29140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:01.662001 29140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:01.662235 29140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:01.662873 29140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:01.663725 29140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:01.663800 29140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.663872 29140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:01.663915 29140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.673866 29140 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37179
I20260810 23:09:01.673887 29271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37179 every 8 connection(s)
I20260810 23:09:01.674834 29140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:09:01.679003 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29140
I20260810 23:09:01.679139 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:09:01.679415 14636 external_mini_cluster.cc:1428] Setting key d2a6fa7a4cdd30135f6abf4b611ea402
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:09:01.690207 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.676686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:42491 (local address 127.14.75.62:42755)
0810 23:09:01.676834 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.676838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.677791 (+ 953us) server_negotiation.cc:407] Connection header received
0810 23:09:01.679214 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.679217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.679278 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.679390 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:01.680926 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.681722 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.683153 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.683391 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.683519 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.686266 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:01.686269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:01.686271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:01.686297 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:01.687842 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.688403 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.688407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.688409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.688497 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.688876 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.688880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.688882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.689086 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:01.689210 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.689836 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.689978 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:09:01.691345 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.677096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:42491)
0810 23:09:01.677631 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.677678 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.678971 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.679583 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.679589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.680019 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:01.680564 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.680607 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.681852 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.681854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.683003 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.683019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.683569 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.683574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.683760 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.684424 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:01.684441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:01.686062 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:01.687980 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.687987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.688005 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.688294 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.688597 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.688600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.688602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.688771 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.689202 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:01.689209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:01.689572 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.689790 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.690186 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:09:01.692708 29272 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:01.692974 29272 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:01.693645 29272 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:01.694980 28937 ts_manager.cc:194] Registered new tserver with Master: 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
I20260810 23:09:01.695600 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:42491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:09:01.711419 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:01.841915 29277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:01.842214 29277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:01.842260 29277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:01.846616 29277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:01.846696 29277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:01.846760 29277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:09:01.852428 29277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:01.853804 29277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:01.854889 29277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:01.865082 29282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:01.865082 29283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:01.865322 29277 server_base.cc:1034] running on GCE node
W20260810 23:09:01.865089 29285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:01.865933 29277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:01.866606 29277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:01.867775 29277 hybrid_clock.cc:648] HybridClock initialized: now 1786403341867739 us; error 49 us; skew 500 ppm
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:01.870855 29277 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:09:01.871922 29277 webserver.cc:460] Webserver started at http://127.14.75.3:34807/ using document root <none> and password file <none>
I20260810 23:09:01.872575 29277 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:01.872646 29277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:01.872911 29277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:01.875213 29277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "852bc9ad18454a1b93368bdea2ecdeea"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "eff272cc82a16b44959fd5dc8bb4ce0f"
server_key_iv: "1b1a7fb5828bda6df0d2383257acf3f3"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.875798 29277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "852bc9ad18454a1b93368bdea2ecdeea"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "eff272cc82a16b44959fd5dc8bb4ce0f"
server_key_iv: "1b1a7fb5828bda6df0d2383257acf3f3"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.879779 29277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 23:09:01.882512 29292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.883883 29277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:01.884097 29277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "852bc9ad18454a1b93368bdea2ecdeea"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "eff272cc82a16b44959fd5dc8bb4ce0f"
server_key_iv: "1b1a7fb5828bda6df0d2383257acf3f3"
server_key_version: "encryptionkey@0"
I20260810 23:09:01.884238 29277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:01.900282 29277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:01.903680 29277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:01.903910 29277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:01.904577 29277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:01.905524 29277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:01.905598 29277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.905668 29277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:01.905719 29277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:01.915432 29277 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:40191
I20260810 23:09:01.915535 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:40191 every 8 connection(s)
I20260810 23:09:01.916517 29277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:09:01.920373 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29277
I20260810 23:09:01.920552 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:09:01.920861 14636 external_mini_cluster.cc:1428] Setting key c5d858e6a88b416ebfb5fff6a19ee425
Aug 10 23:09:01 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403341, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:01.931752 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.918559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:56107 (local address 127.14.75.62:42755)
0810 23:09:01.918725 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.918729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.919671 (+ 942us) server_negotiation.cc:407] Connection header received
0810 23:09:01.921583 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.921587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.921652 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.921766 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:01.923124 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.923673 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.924953 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.925187 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.925301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.928191 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:01.928195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:01.928197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:01.928225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:01.929634 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.930178 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.930181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.930182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.930262 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.930578 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.930580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.930581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.930778 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:01.930856 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.931488 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.931609 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:09:01.932639 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.919015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:56107)
0810 23:09:01.919503 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.919551 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.921267 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.921978 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.921987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.922434 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:01.922958 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.922973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.923818 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.923821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.924792 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.924811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.925355 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.925359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.925589 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.926266 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:01.926283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:01.927927 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:01.929757 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.929762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.929777 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.930068 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.930343 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:01.930345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:01.930347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:01.930496 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:01.930901 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:01.930907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:01.931229 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.931407 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.931696 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260810 23:09:01.933944 29408 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:01.934291 29408 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:01.934990 29408 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:01.936131 28937 ts_manager.cc:194] Registered new tserver with Master: 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191)
I20260810 23:09:01.936688 28937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:56107
I20260810 23:09:01.947515 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:09:01.959000 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.950207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.14.75.62:42755)
0810 23:09:01.950363 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.950367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.950626 (+ 259us) server_negotiation.cc:407] Connection header received
0810 23:09:01.950768 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.950770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.950819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.950918 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:01.951807 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.952718 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.953968 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.954203 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.954296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.955751 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:01.955753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:01.955755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:01.955780 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:01.957301 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.957778 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.957781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.957782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.957870 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:01.958135 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:01.958138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:01.958139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:01.958369 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:01.958472 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.958680 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.958783 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:09:01.962146 28937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:09:01.965265 28937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:09:01.981344 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.974645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40191 (local address 127.0.0.1:46360)
0810 23:09:01.975633 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.975694 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.975866 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.976466 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.976472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.976495 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:01.976644 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.976651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978227 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.978230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.979888 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.979899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.980743 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.980746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.980866 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.980902 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.981016 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.981155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":217,"thread_start_us":106,"threads_started":1}
I20260810 23:09:01.981684 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.975845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.14.75.2:37179)
0810 23:09:01.976226 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.976236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.976284 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:09:01.976364 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.976373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.976561 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.976728 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:01.976982 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978138 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.979984 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.980795 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.980890 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.981018 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.981348 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.981500 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":162,"threads_started":1}
I20260810 23:09:01.982028 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.975239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.14.75.1:33455)
0810 23:09:01.977039 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.977044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.977061 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:09:01.977125 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.977129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.977257 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.977405 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:01.977667 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978720 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.979814 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.980663 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.980776 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.980930 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.981794 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.981903 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":88,"threads_started":1}
I20260810 23:09:01.982254 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.974831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.14.75.3:40191)
0810 23:09:01.975387 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:01.975400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:01.975784 (+ 384us) server_negotiation.cc:407] Connection header received
0810 23:09:01.976071 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:01.976077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:01.976289 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:01.976465 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:01.976782 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978048 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.980017 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.980601 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.980741 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.980906 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:01.981059 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:09:01.982058 (+ 999us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":189,"threads_started":1}
I20260810 23:09:01.983153 29423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.975056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:33455 (local address 127.0.0.1:45040)
0810 23:09:01.975672 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.975720 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.975851 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.977385 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.977388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.977444 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:01.977526 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.977536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978832 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.978838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.979679 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.979689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.981006 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.981010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.981102 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.981188 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.981264 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.981357 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260810 23:09:01.983323 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:01.975420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37179 (local address 127.0.0.1:34114)
0810 23:09:01.975871 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:01.975883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:01.975965 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:01.976704 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:01.976707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:01.976725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:01.976839 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.976848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.978277 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.978279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.979846 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:01.979858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:01.981069 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:01.981072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:01.981189 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:01.981206 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:09:01.981269 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:09:01.982264 (+ 995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":1003,"thread_start_us":95,"threads_started":1}
I20260810 23:09:01.985491 29341 tablet_service.cc:1467] Processing CreateTablet for tablet e2e8917e84c849a5b4fefab784c89e9a (DEFAULT_TABLE table=test-table [id=a946474ab4644d5993e2263d73e17641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:01.985716 29203 tablet_service.cc:1467] Processing CreateTablet for tablet e2e8917e84c849a5b4fefab784c89e9a (DEFAULT_TABLE table=test-table [id=a946474ab4644d5993e2263d73e17641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:01.986017 29066 tablet_service.cc:1467] Processing CreateTablet for tablet e2e8917e84c849a5b4fefab784c89e9a (DEFAULT_TABLE table=test-table [id=a946474ab4644d5993e2263d73e17641]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:01.986757 29341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e8917e84c849a5b4fefab784c89e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:01.986769 29203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e8917e84c849a5b4fefab784c89e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:01.986917 29066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e8917e84c849a5b4fefab784c89e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:01.993644 29425 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Bootstrap starting.
I20260810 23:09:01.994879 29426 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Bootstrap starting.
I20260810 23:09:01.995239 29428 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Bootstrap starting.
I20260810 23:09:01.996873 29426 tablet_bootstrap.cc:654] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:01.997287 29425 tablet_bootstrap.cc:654] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:01.997287 29428 tablet_bootstrap.cc:654] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:01.997805 29426 log.cc:824] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:01.998238 29425 log.cc:824] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:01.998576 29428 log.cc:824] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:01.999933 29426 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: No bootstrap required, opened a new log
I20260810 23:09:02.000160 29426 ts_tablet_manager.cc:1397] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:09:02.000299 29425 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: No bootstrap required, opened a new log
I20260810 23:09:02.000488 29428 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: No bootstrap required, opened a new log
I20260810 23:09:02.000497 29425 ts_tablet_manager.cc:1397] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:09:02.000674 29428 ts_tablet_manager.cc:1397] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:02.004321 29426 raft_consensus.cc:348] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.004640 29426 raft_consensus.cc:374] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:02.004524 29428 raft_consensus.cc:348] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.004524 29425 raft_consensus.cc:348] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.004730 29426 raft_consensus.cc:729] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365b6146a7a6479f83acb9718a1a3879, State: Initialized, Role: FOLLOWER
I20260810 23:09:02.004791 29425 raft_consensus.cc:374] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:02.004789 29428 raft_consensus.cc:374] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:02.004844 29425 raft_consensus.cc:729] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983e2e7717d24f87bdfb37a18661757e, State: Initialized, Role: FOLLOWER
I20260810 23:09:02.004853 29428 raft_consensus.cc:729] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852bc9ad18454a1b93368bdea2ecdeea, State: Initialized, Role: FOLLOWER
I20260810 23:09:02.005265 29426 consensus_queue.cc:260] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.005301 29428 consensus_queue.cc:260] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.005301 29425 consensus_queue.cc:260] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.006008 29135 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
I20260810 23:09:02.006350 29426 ts_tablet_manager.cc:1428] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:09:02.006364 29425 ts_tablet_manager.cc:1428] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:09:02.006726 29272 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
I20260810 23:09:02.007332 29428 ts_tablet_manager.cc:1428] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:09:02.007372 29408 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
W20260810 23:09:02.167591 29409 tablet.cc:2369] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:09:02.176043 29273 tablet.cc:2369] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:09:02.190713 29432 raft_consensus.cc:482] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:02.190977 29432 raft_consensus.cc:504] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.192317 29432 leader_election.cc:288] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
W20260810 23:09:02.196507 29136 tablet.cc:2369] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:09:02.196691 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.192717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:40437 (local address 127.14.75.3:40191)
0810 23:09:02.192913 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.192918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.193030 (+ 112us) server_negotiation.cc:407] Connection header received
0810 23:09:02.193213 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.193222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.193302 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.193397 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:02.193814 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.194442 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.195654 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.196116 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.196198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.196279 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.196499 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.196567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:09:02.196728 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.192612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40191 (local address 127.14.75.1:40437)
0810 23:09:02.192923 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.192937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:02.193090 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:02.193448 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:02.193451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:02.193480 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:02.193671 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.193678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.194565 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.194569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.195505 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.195515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.196241 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.196247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.196357 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.196386 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:09:02.196480 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:09:02.196552 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260810 23:09:02.196785 29424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.193233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:36729 (local address 127.14.75.2:37179)
0810 23:09:02.193458 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.193463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.193570 (+ 107us) server_negotiation.cc:407] Connection header received
0810 23:09:02.193680 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.193684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.193743 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.193830 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:02.194113 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.194696 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.195777 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.196152 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.196287 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.196354 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.196592 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.196669 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:09:02.197127 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.193081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37179 (local address 127.14.75.1:36729)
0810 23:09:02.193466 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.193480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:02.193602 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:02.193855 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:02.193857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:02.193874 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:02.193979 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.193986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.194815 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.194818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.195655 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.195664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.196320 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.196326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.196436 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.196458 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:09:02.196914 (+ 456us) client_negotiation.cc:240] Negotiation successful
0810 23:09:02.196984 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260810 23:09:02.197562 29361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e8917e84c849a5b4fefab784c89e9a" candidate_uuid: "365b6146a7a6479f83acb9718a1a3879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852bc9ad18454a1b93368bdea2ecdeea" is_pre_election: true
I20260810 23:09:02.197890 29361 raft_consensus.cc:2454] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365b6146a7a6479f83acb9718a1a3879 in term 0.
I20260810 23:09:02.197919 29223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e8917e84c849a5b4fefab784c89e9a" candidate_uuid: "365b6146a7a6479f83acb9718a1a3879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983e2e7717d24f87bdfb37a18661757e" is_pre_election: true
I20260810 23:09:02.198204 29223 raft_consensus.cc:2454] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 365b6146a7a6479f83acb9718a1a3879 in term 0.
I20260810 23:09:02.198423 29019 leader_election.cc:302] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879, 852bc9ad18454a1b93368bdea2ecdeea; no voters:
I20260810 23:09:02.198693 29432 raft_consensus.cc:2781] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:09:02.198769 29432 raft_consensus.cc:482] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:09:02.198820 29432 raft_consensus.cc:3037] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:02.199859 29432 raft_consensus.cc:504] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.200213 29432 leader_election.cc:288] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 1 election: Requested vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
I20260810 23:09:02.200591 29361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e8917e84c849a5b4fefab784c89e9a" candidate_uuid: "365b6146a7a6479f83acb9718a1a3879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852bc9ad18454a1b93368bdea2ecdeea"
I20260810 23:09:02.200659 29223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e8917e84c849a5b4fefab784c89e9a" candidate_uuid: "365b6146a7a6479f83acb9718a1a3879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983e2e7717d24f87bdfb37a18661757e"
I20260810 23:09:02.200740 29361 raft_consensus.cc:3037] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:02.200816 29223 raft_consensus.cc:3037] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:02.201977 29361 raft_consensus.cc:2454] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365b6146a7a6479f83acb9718a1a3879 in term 1.
I20260810 23:09:02.202133 29223 raft_consensus.cc:2454] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 365b6146a7a6479f83acb9718a1a3879 in term 1.
I20260810 23:09:02.202291 29019 leader_election.cc:302] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879, 852bc9ad18454a1b93368bdea2ecdeea; no voters:
I20260810 23:09:02.202500 29432 raft_consensus.cc:2781] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:02.202716 29432 raft_consensus.cc:686] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 LEADER]: Becoming Leader. State: Replica: 365b6146a7a6479f83acb9718a1a3879, State: Running, Role: LEADER
I20260810 23:09:02.203049 29432 consensus_queue.cc:237] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.206380 28935 catalog_manager.cc:5495] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 reported cstate change: term changed from 0 to 1, leader changed from <none> to 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "365b6146a7a6479f83acb9718a1a3879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } health_report { overall_health: HEALTHY } } }
I20260810 23:09:02.217317 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.213853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.14.75.1:33455)
0810 23:09:02.213980 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.213984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.213997 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:09:02.214165 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.214170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.214240 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.214339 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:02.214585 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.215087 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.215969 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.216184 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.216273 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.216527 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:02.216944 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:09:02.217034 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.217089 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.217137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:09:02.223424 29361 raft_consensus.cc:1261] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Refusing update from remote peer 365b6146a7a6479f83acb9718a1a3879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:02.223424 29223 raft_consensus.cc:1261] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Refusing update from remote peer 365b6146a7a6479f83acb9718a1a3879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:02.224234 29432 consensus_queue.cc:1035] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:02.224475 29437 consensus_queue.cc:1035] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:02.245285 28935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:02.245517 28935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:02.248090 28935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Sending DeleteTablet for 3 replicas of tablet e2e8917e84c849a5b4fefab784c89e9a
I20260810 23:09:02.248991 29066 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e8917e84c849a5b4fefab784c89e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45040
I20260810 23:09:02.248991 29341 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e8917e84c849a5b4fefab784c89e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46360
I20260810 23:09:02.249609 29451 tablet_replica.cc:330] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: stopping tablet replica
I20260810 23:09:02.249634 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29002
I20260810 23:09:02.249750 29203 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e8917e84c849a5b4fefab784c89e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 23:09:02.250069 29451 raft_consensus.cc:2229] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:09:02.250561 29452 tablet_replica.cc:330] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: stopping tablet replica
I20260810 23:09:02.250888 29452 raft_consensus.cc:2229] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:02.251291 29452 raft_consensus.cc:2258] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:02.251528 29454 tablet_replica.cc:330] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: stopping tablet replica
I20260810 23:09:02.251823 29454 raft_consensus.cc:2229] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:02.252566 29454 raft_consensus.cc:2258] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:02.253561 29452 ts_tablet_manager.cc:1905] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:02.253827 29454 ts_tablet_manager.cc:1905] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:02.256060 29454 ts_tablet_manager.cc:1918] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:02.256206 29454 log.cc:1196] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/e2e8917e84c849a5b4fefab784c89e9a
I20260810 23:09:02.256281 29452 ts_tablet_manager.cc:1918] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:02.256385 29452 log.cc:1196] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/e2e8917e84c849a5b4fefab784c89e9a
I20260810 23:09:02.256570 29454 ts_tablet_manager.cc:1939] T e2e8917e84c849a5b4fefab784c89e9a P 983e2e7717d24f87bdfb37a18661757e: Deleting consensus metadata
I20260810 23:09:02.256708 29452 ts_tablet_manager.cc:1939] T e2e8917e84c849a5b4fefab784c89e9a P 852bc9ad18454a1b93368bdea2ecdeea: Deleting consensus metadata
I20260810 23:09:02.257934 28922 catalog_manager.cc:4841] TS 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): tablet e2e8917e84c849a5b4fefab784c89e9a (table test-table [id=a946474ab4644d5993e2263d73e17641]) successfully deleted
I20260810 23:09:02.258189 28922 catalog_manager.cc:4841] TS 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): tablet e2e8917e84c849a5b4fefab784c89e9a (table test-table [id=a946474ab4644d5993e2263d73e17641]) successfully deleted
W20260810 23:09:02.258612 28924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:33455 (error 108)
I20260810 23:09:02.258927 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29140
I20260810 23:09:02.260671 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.260329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:33455 (local address 127.0.0.1:45054)
0810 23:09:02.260523 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.260595 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 23:09:02.260946 28924 catalog_manager.cc:4584] TS 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2e8917e84c849a5b4fefab784c89e9a: Network error: Client connection negotiation failed: client connection to 127.14.75.1:33455: connect: Connection refused (error 111)
I20260810 23:09:02.265926 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29277
I20260810 23:09:02.273134 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 28905
I20260810 23:09:02.280499 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--webserver_interface=127.14.75.62
--webserver_port=34553
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:02.396862 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:02.397167 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:02.397210 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:02.401439 29456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:09:02.401517 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:02.401537 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:02.401556 29456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:09:02.401571 29456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:09:02.407289 29456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=34553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:02.408506 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:02.409685 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:09:02.425283 29456 server_base.cc:1034] running on GCE node
W20260810 23:09:02.425230 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:02.425257 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:02.425503 29466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:02.426247 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:02.427531 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:02.428741 29456 hybrid_clock.cc:648] HybridClock initialized: now 1786403342428723 us; error 33 us; skew 500 ppm
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:02.431900 29456 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:09:02.432863 29456 webserver.cc:460] Webserver started at http://127.14.75.62:34553/ using document root <none> and password file <none>
I20260810 23:09:02.433526 29456 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:02.433616 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:02.438076 29456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:09:02.440383 29473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:02.441905 29456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:02.442027 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "ffda23007075428c90158bfe73ffeccf"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "c7ba2352c78ebf773f9945198840aa03"
server_key_iv: "3ba98025597fa4cf62e152486c1cfc34"
server_key_version: "encryptionkey@0"
I20260810 23:09:02.442657 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:02.461853 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:02.465801 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:02.466033 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:02.473819 29456 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42755
I20260810 23:09:02.474141 29527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42755 every 8 connection(s)
I20260810 23:09:02.475328 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:09:02.478031 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29456
I20260810 23:09:02.478492 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:33455
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=36493
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:02.480441 29528 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:09:02.483115 29528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap starting.
I20260810 23:09:02.507851 29528 log.cc:824] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:02.516973 29528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:02.517498 29528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap complete.
I20260810 23:09:02.521812 29528 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:02.522297 29528 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Initialized, Role: FOLLOWER
I20260810 23:09:02.522867 29528 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:02.523025 29528 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:09:02.523135 29528 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:09:02.523260 29528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:09:02.526162 29528 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:02.526576 29528 leader_election.cc:302] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffda23007075428c90158bfe73ffeccf; no voters:
I20260810 23:09:02.526966 29528 leader_election.cc:288] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:09:02.527439 29532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:09:02.527721 29532 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 LEADER]: Becoming Leader. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Running, Role: LEADER
I20260810 23:09:02.528038 29532 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:02.528412 29528 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:09:02.529049 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffda23007075428c90158bfe73ffeccf. Latest consensus state: current_term: 2 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:02.529181 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:02.529625 29538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:09:02.529654 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:02.529911 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:02.534730 29538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:09:02.535426 29538 catalog_manager.cc:1259] Loaded cluster ID: 362d3d85e4f74ab1af0d443de73780ef
I20260810 23:09:02.535625 29538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:09:02.537596 29538 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:09:02.542071 29538 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:09:02.548379 29538 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:09:02.549261 29538 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Loaded TSK: 0
I20260810 23:09:02.550549 29538 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:09:02.612684 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:02.613020 29530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:02.613099 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:02.617466 29530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:02.617573 29530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:02.617671 29530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:09:02.623831 29530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:33455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=36493
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:02.624962 29530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:02.626156 29530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:02.636580 29556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:02.636567 29558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:02.636802 29530 server_base.cc:1034] running on GCE node
W20260810 23:09:02.636564 29555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:02.637534 29530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:02.638468 29530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:02.639698 29530 hybrid_clock.cc:648] HybridClock initialized: now 1786403342639659 us; error 54 us; skew 500 ppm
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:02.643478 29530 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:09:02.644559 29530 webserver.cc:460] Webserver started at http://127.14.75.1:36493/ using document root <none> and password file <none>
I20260810 23:09:02.645160 29530 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:02.645242 29530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:02.650187 29530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:02.652664 29566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:02.654167 29530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:02.654356 29530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "365b6146a7a6479f83acb9718a1a3879"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "4a6af9286d415e2acf3456c41dc60ce1"
server_key_iv: "5368dba2f77fd53dd657ce4cbb624675"
server_key_version: "encryptionkey@0"
I20260810 23:09:02.654982 29530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:02.675518 29530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:02.679399 29530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:02.679651 29530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:02.680313 29530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:02.681545 29573 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:09:02.686393 29530 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:09:02.686518 29530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 23:09:02.686614 29530 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:09:02.688508 29530 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:09:02.688601 29530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:09:02.688895 29573 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Bootstrap starting.
I20260810 23:09:02.699767 29530 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:33455
I20260810 23:09:02.699846 29683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:33455 every 8 connection(s)
I20260810 23:09:02.700834 29530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:09:02.707551 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29530
I20260810 23:09:02.708084 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:37179
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=34299
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:02.710397 29573 log.cc:824] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:02.721143 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.702957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:46415)
0810 23:09:02.703849 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.703911 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:02.705234 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:02.706512 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:02.706521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:02.707052 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:02.707847 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.707877 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.709982 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.709985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.711506 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.711522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.712598 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.712602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.712755 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.713513 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:02.713530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:02.715223 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:02.717712 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.717720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.717741 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.718010 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.718430 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.718435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.718437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.718583 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.719243 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:02.719255 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:02.719534 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 23:09:02.719739 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:09:02.720135 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":169,"threads_started":1}
I20260810 23:09:02.721117 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.703064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:46415 (local address 127.14.75.62:42755)
0810 23:09:02.703509 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.703529 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.704069 (+ 540us) server_negotiation.cc:407] Connection header received
0810 23:09:02.705726 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.705737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.706056 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.706356 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:02.708083 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.709751 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.711674 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.712428 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.712616 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.715463 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:02.715468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:02.715481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:02.715513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:02.717504 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.718164 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.718171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.718177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.718320 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.718707 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.718711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.718713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.719105 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:02.719279 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.719860 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.720146 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":154,"threads_started":1}
I20260810 23:09:02.721570 29573 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:02.722283 29573 tablet_bootstrap.cc:492] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Bootstrap complete.
I20260810 23:09:02.723990 29573 ts_tablet_manager.cc:1397] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260810 23:09:02.724030 29684 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:02.724296 29684 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:02.724824 29684 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:02.727120 29573 raft_consensus.cc:348] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.727612 29492 ts_manager.cc:194] Registered new tserver with Master: 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455)
I20260810 23:09:02.727677 29573 raft_consensus.cc:729] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365b6146a7a6479f83acb9718a1a3879, State: Initialized, Role: FOLLOWER
I20260810 23:09:02.728281 29573 consensus_queue.cc:260] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:02.729478 29573 ts_tablet_manager.cc:1428] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 23:09:02.731778 29492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:46415
I20260810 23:09:02.734476 29684 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:09:02.742993 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.731173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.14.75.1:33455)
0810 23:09:02.731566 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.731578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.732018 (+ 440us) server_negotiation.cc:407] Connection header received
0810 23:09:02.732246 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.732251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.732457 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.732615 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:02.733563 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.734600 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.735921 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.736432 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.736520 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.739141 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:02.739152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:02.739180 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:02.739203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:02.740821 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.741376 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.741382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.741384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.741520 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.741830 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.741836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.741840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.742102 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:02.742259 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.742621 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.742804 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260810 23:09:02.742993 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.731076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:33455 (local address 127.0.0.1:45070)
0810 23:09:02.731866 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.731920 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:02.732077 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:02.732763 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:02.732769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:02.733226 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:02.733397 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.733433 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.734741 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.734745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.735779 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.735791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.736559 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.736563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.736703 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.737347 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:02.737375 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:02.738992 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:02.740952 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.740963 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.740969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.741241 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.741620 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.741631 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.741633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.741735 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.742234 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:02.742240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:02.742378 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 23:09:02.742537 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:09:02.742813 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":115,"threads_started":1}
I20260810 23:09:02.744290 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e8917e84c849a5b4fefab784c89e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:02 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45070
I20260810 23:09:02.745250 29693 tablet_replica.cc:330] stopping tablet replica
I20260810 23:09:02.745615 29693 raft_consensus.cc:2229] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:02.745893 29693 raft_consensus.cc:2258] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:02.746948 29693 ts_tablet_manager.cc:1905] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:02.751271 29693 ts_tablet_manager.cc:1918] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:02.751371 29693 log.cc:1196] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/e2e8917e84c849a5b4fefab784c89e9a
I20260810 23:09:02.751690 29693 ts_tablet_manager.cc:1939] T e2e8917e84c849a5b4fefab784c89e9a P 365b6146a7a6479f83acb9718a1a3879: Deleting consensus metadata
I20260810 23:09:02.753523 29477 catalog_manager.cc:4841] TS 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455): tablet e2e8917e84c849a5b4fefab784c89e9a (table test-table [id=a946474ab4644d5993e2263d73e17641]) successfully deleted
W20260810 23:09:02.836431 29688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:02.836788 29688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:02.836887 29688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:02.841248 29688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:02.841378 29688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:02.841502 29688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:09:02.847522 29688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:37179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=34299
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:02.848794 29688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:02.849923 29688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:02.860242 29701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:02.860278 29688 server_base.cc:1034] running on GCE node
W20260810 23:09:02.860239 29699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:02.860232 29698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:02.860994 29688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:02.861812 29688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:02.863013 29688 hybrid_clock.cc:648] HybridClock initialized: now 1786403342862993 us; error 42 us; skew 500 ppm
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:02.866191 29688 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:09:02.867177 29688 webserver.cc:460] Webserver started at http://127.14.75.2:34299/ using document root <none> and password file <none>
I20260810 23:09:02.867872 29688 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:02.867975 29688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:02.872191 29688 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:09:02.874372 29708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:02.875638 29688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:09:02.875808 29688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "983e2e7717d24f87bdfb37a18661757e"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "f88cd05066f71a39754095614b348e28"
server_key_iv: "64d097c66da00935bb7fd8045cd73458"
server_key_version: "encryptionkey@0"
I20260810 23:09:02.876382 29688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:02.905606 29688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:02.908874 29688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:02.909113 29688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:02.909798 29688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:02.910681 29688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:02.910768 29688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:02.910842 29688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:02.910885 29688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:02.920537 29688 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37179
I20260810 23:09:02.920586 29825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37179 every 8 connection(s)
I20260810 23:09:02.921634 29688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:09:02.928965 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29688
I20260810 23:09:02.929500 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:40191
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=34807
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:09:02 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403342, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:02.937584 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.923669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:48395 (local address 127.14.75.62:42755)
0810 23:09:02.923820 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:02.923824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:02.924813 (+ 989us) server_negotiation.cc:407] Connection header received
0810 23:09:02.926324 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:02.926328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:02.926389 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:02.926476 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:02.927830 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.928382 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.930380 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.930651 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.930749 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.933684 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:02.933688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:02.933690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:02.933721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:02.935284 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.935858 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.935862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.935863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.935945 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:02.936329 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:02.936332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:02.936334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:02.936531 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:02.936651 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:02.937210 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 23:09:02.937358 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:09:02.938526 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:02.924088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:48395)
0810 23:09:02.924632 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:02.924680 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:02.926092 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:02.926657 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:02.926664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:02.927084 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:02.927596 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.927610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.928535 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.928538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.930207 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:02.930223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:02.930833 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:02.930840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:02.931105 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:02.931867 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:02.931885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:02.933471 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:02.935421 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.935427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.935445 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.935734 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.936040 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:02.936045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:02.936047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:02.936222 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:02.936649 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:02.936656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:02.936932 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 23:09:02.937155 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 23:09:02.937553 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":132,"threads_started":1}
I20260810 23:09:02.939790 29826 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:02.940056 29826 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:02.940579 29826 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:02.941696 29492 ts_manager.cc:194] Registered new tserver with Master: 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
I20260810 23:09:02.942291 29492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:48395
W20260810 23:09:03.048899 29829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:03.049247 29829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:03.049329 29829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:03.053812 29829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:03.053889 29829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:03.053982 29829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:09:03.059828 29829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:40191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=34807
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:03.061041 29829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:03.062180 29829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:03.072579 29838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.072579 29835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.072618 29836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:03.073006 29829 server_base.cc:1034] running on GCE node
I20260810 23:09:03.073479 29829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:03.074177 29829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:03.075354 29829 hybrid_clock.cc:648] HybridClock initialized: now 1786403343075337 us; error 27 us; skew 500 ppm
Aug 10 23:09:03 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:03.078495 29829 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:09:03.079432 29829 webserver.cc:460] Webserver started at http://127.14.75.3:34807/ using document root <none> and password file <none>
I20260810 23:09:03.080025 29829 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:03.080130 29829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:03.084193 29829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:09:03.086442 29846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.087822 29829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:03.087930 29829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "852bc9ad18454a1b93368bdea2ecdeea"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "eff272cc82a16b44959fd5dc8bb4ce0f"
server_key_iv: "1b1a7fb5828bda6df0d2383257acf3f3"
server_key_version: "encryptionkey@0"
I20260810 23:09:03.088546 29829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:03.109124 29829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:03.112267 29829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:03.112478 29829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:03.113101 29829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:03.114040 29829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:03.114095 29829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.114161 29829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:03.114193 29829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.123400 29829 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:40191
I20260810 23:09:03.123431 29962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:40191 every 8 connection(s)
I20260810 23:09:03.124504 29829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:09:03.127832 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 29829
Aug 10 23:09:03 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:03.137895 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.128615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37026 (local address 127.14.75.62:42755)
0810 23:09:03.128900 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.128904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.129023 (+ 119us) server_negotiation.cc:407] Connection header received
0810 23:09:03.129190 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.129195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.129245 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.129396 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:03.130278 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.131227 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.132122 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.132371 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.132508 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.134119 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:03.134123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:03.134126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:03.134152 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:03.135937 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:03.136475 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:03.136477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:03.136479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:03.136559 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:03.136876 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:03.136878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:03.136879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:03.137051 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:03.137279 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.137529 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.137660 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":104,"threads_started":1}
I20260810 23:09:03.139140 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.126374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:46731 (local address 127.14.75.62:42755)
0810 23:09:03.126551 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.126556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.127415 (+ 859us) server_negotiation.cc:407] Connection header received
0810 23:09:03.129134 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.129138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.129214 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.129304 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:03.130748 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.131330 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.132467 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.132682 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.132801 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.135194 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:03.135203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:03.135205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:03.135237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:03.136863 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:03.137355 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:03.137360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:03.137361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:03.137463 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:03.137883 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:03.137887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:03.137888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:03.138133 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:03.138273 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.138883 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.138997 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:09:03.140095 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.126781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:46731)
0810 23:09:03.127253 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.127298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.128913 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.129536 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:03.129543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:03.129984 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:03.130596 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.130612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.131455 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.131458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.132333 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.132342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.132844 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.132848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.133016 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.133569 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:03.133586 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:03.135025 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:03.136955 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:03.136961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:03.136984 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:03.137249 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:03.137556 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:03.137563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:03.137566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:03.137706 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:03.138278 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:03.138287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:03.138629 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 23:09:03.138841 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 23:09:03.139192 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260810 23:09:03.141204 29963 heartbeater.cc:347] Connected to a master server at 127.14.75.62:42755
I20260810 23:09:03.141520 29963 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:03.142023 29963 heartbeater.cc:511] Master 127.14.75.62:42755 requested a full tablet report, sending...
I20260810 23:09:03.143047 29492 ts_manager.cc:194] Registered new tserver with Master: 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191)
I20260810 23:09:03.143612 29492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:46731
I20260810 23:09:03.151829 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:09:03.156172 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.152520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.14.75.62:42755)
0810 23:09:03.152660 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.152665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.152847 (+ 182us) server_negotiation.cc:407] Connection header received
0810 23:09:03.152987 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.152989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.153099 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.153199 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:03.153475 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.153982 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.154826 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.155036 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.155113 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.155385 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:03.155784 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:09:03.155875 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.155928 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.155978 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:09:03.156867 29492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:09:03.160346 29492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:09:03.168619 29616 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 (DEFAULT_TABLE table=test-table [id=e94e2eff5d5f452c8eb56056cfa416f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:03.169354 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:03.172307 29573 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Bootstrap starting.
I20260810 23:09:03.173280 29573 tablet_bootstrap.cc:654] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:03.174451 29691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.167653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40191 (local address 127.0.0.1:46366)
0810 23:09:03.167804 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.167822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.167909 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.169442 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:03.169445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:03.169466 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:03.169568 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.169574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.171209 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.171212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.172647 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.172659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.173914 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.173918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.174037 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.174097 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:09:03.174194 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:09:03.174304 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":133}
I20260810 23:09:03.174518 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.168367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.14.75.3:40191)
0810 23:09:03.168927 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.168940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.168963 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:09:03.169043 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.169048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.169246 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.169739 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:03.169901 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.171027 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.172781 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.173766 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.173899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.174072 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.174230 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.174356 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":210,"threads_started":1}
I20260810 23:09:03.174728 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.167976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37179 (local address 127.0.0.1:34130)
0810 23:09:03.168267 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.168285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.168376 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.169311 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:03.169314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:03.169333 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:03.169489 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.169498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.171139 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.171144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.173159 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.173170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.174340 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.174343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.174428 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.174450 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:09:03.174502 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:09:03.174575 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":586,"thread_start_us":138,"threads_started":1}
I20260810 23:09:03.175251 29573 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: No bootstrap required, opened a new log
I20260810 23:09:03.175383 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.168412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.14.75.2:37179)
0810 23:09:03.168782 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.168802 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.168849 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:09:03.168936 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.168942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.169154 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.169328 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:03.169877 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.170999 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.173462 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.174211 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.174880 (+ 669us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.175071 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.175140 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.175246 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260810 23:09:03.175383 29573 ts_tablet_manager.cc:1397] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:09:03.175908 29573 raft_consensus.cc:348] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.176040 29573 raft_consensus.cc:374] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:03.176090 29573 raft_consensus.cc:729] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365b6146a7a6479f83acb9718a1a3879, State: Initialized, Role: FOLLOWER
I20260810 23:09:03.176215 29573 consensus_queue.cc:260] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.176458 29573 ts_tablet_manager.cc:1428] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:09:03.178207 29759 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 (DEFAULT_TABLE table=test-table [id=e94e2eff5d5f452c8eb56056cfa416f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:03.178210 29895 tablet_service.cc:1467] Processing CreateTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 (DEFAULT_TABLE table=test-table [id=e94e2eff5d5f452c8eb56056cfa416f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:03.179368 29759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:03.179371 29895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:03.186388 29977 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Bootstrap starting.
I20260810 23:09:03.186949 29976 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Bootstrap starting.
I20260810 23:09:03.188688 29976 tablet_bootstrap.cc:654] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:03.188956 29977 tablet_bootstrap.cc:654] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:03.189888 29977 log.cc:824] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:03.190117 29976 log.cc:824] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:03.192358 29976 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: No bootstrap required, opened a new log
I20260810 23:09:03.192358 29977 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: No bootstrap required, opened a new log
I20260810 23:09:03.192574 29976 ts_tablet_manager.cc:1397] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:03.192574 29977 ts_tablet_manager.cc:1397] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:03.196780 29976 raft_consensus.cc:348] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.196780 29977 raft_consensus.cc:348] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.197110 29976 raft_consensus.cc:374] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:03.197110 29977 raft_consensus.cc:374] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:03.197167 29976 raft_consensus.cc:729] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852bc9ad18454a1b93368bdea2ecdeea, State: Initialized, Role: FOLLOWER
I20260810 23:09:03.197178 29977 raft_consensus.cc:729] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983e2e7717d24f87bdfb37a18661757e, State: Initialized, Role: FOLLOWER
I20260810 23:09:03.197711 29976 consensus_queue.cc:260] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.197711 29977 consensus_queue.cc:260] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.198485 29826 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
I20260810 23:09:03.198747 29963 heartbeater.cc:503] Master 127.14.75.62:42755 was elected leader, sending a full tablet report...
I20260810 23:09:03.198877 29976 ts_tablet_manager.cc:1428] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:09:03.198876 29977 ts_tablet_manager.cc:1428] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:09:03.201875 29980 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:03.202070 29980 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
W20260810 23:09:03.202600 29685 tablet.cc:2369] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:09:03.203271 29980 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455)
I20260810 23:09:03.208050 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.203522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40191 (local address 127.14.75.2:44911)
0810 23:09:03.203721 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.203738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.203822 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.204085 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:03.204089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:03.204114 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:03.204252 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.204258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.205288 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.205290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.206644 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.206655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.207586 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.207590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.207703 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.207730 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:09:03.207811 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:09:03.207896 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":56}
I20260810 23:09:03.208204 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.203659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:45265 (local address 127.14.75.1:33455)
0810 23:09:03.203768 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.203773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.204189 (+ 416us) server_negotiation.cc:407] Connection header received
0810 23:09:03.204249 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.204251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.204322 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.204429 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:03.204654 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.205386 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.206791 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.207496 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.207602 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.207774 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.208014 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.208095 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 23:09:03.208220 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.203522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:33455 (local address 127.14.75.2:45265)
0810 23:09:03.204077 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.204089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.204179 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.204441 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:03.204443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:03.204458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:03.204545 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.204549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.205521 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.205524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.206665 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:03.206671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.207728 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.207733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:03.207855 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.207884 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:09:03.207983 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:09:03.208064 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":26,"thread_start_us":135,"threads_started":1}
I20260810 23:09:03.208613 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.203596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:44911 (local address 127.14.75.3:40191)
0810 23:09:03.203779 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.203785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.203800 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:09:03.203931 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.203933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.203972 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.204051 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:03.204364 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.205181 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.206791 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.207444 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.207575 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.207644 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.208419 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.208497 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 23:09:03.208887 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d8a319ee94dc9b5e7d08012b6fd58" candidate_uuid: "983e2e7717d24f87bdfb37a18661757e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365b6146a7a6479f83acb9718a1a3879" is_pre_election: true
I20260810 23:09:03.209255 29637 raft_consensus.cc:2454] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983e2e7717d24f87bdfb37a18661757e in term 0.
I20260810 23:09:03.209227 29916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d8a319ee94dc9b5e7d08012b6fd58" candidate_uuid: "983e2e7717d24f87bdfb37a18661757e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852bc9ad18454a1b93368bdea2ecdeea" is_pre_election: true
I20260810 23:09:03.209537 29916 raft_consensus.cc:2454] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983e2e7717d24f87bdfb37a18661757e in term 0.
I20260810 23:09:03.209779 29713 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879, 983e2e7717d24f87bdfb37a18661757e; no voters:
I20260810 23:09:03.210081 29980 raft_consensus.cc:2781] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:09:03.210187 29980 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:09:03.210251 29980 raft_consensus.cc:3037] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:03.211331 29980 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.211710 29980 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [CANDIDATE]: Term 1 election: Requested vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455)
I20260810 23:09:03.212066 29916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d8a319ee94dc9b5e7d08012b6fd58" candidate_uuid: "983e2e7717d24f87bdfb37a18661757e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852bc9ad18454a1b93368bdea2ecdeea"
I20260810 23:09:03.212248 29916 raft_consensus.cc:3037] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:03.212270 29637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f4d8a319ee94dc9b5e7d08012b6fd58" candidate_uuid: "983e2e7717d24f87bdfb37a18661757e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "365b6146a7a6479f83acb9718a1a3879"
I20260810 23:09:03.212430 29637 raft_consensus.cc:3037] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:03.213496 29916 raft_consensus.cc:2454] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983e2e7717d24f87bdfb37a18661757e in term 1.
I20260810 23:09:03.213596 29637 raft_consensus.cc:2454] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983e2e7717d24f87bdfb37a18661757e in term 1.
I20260810 23:09:03.213830 29711 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e; no voters:
I20260810 23:09:03.214061 29980 raft_consensus.cc:2781] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:03.214264 29980 raft_consensus.cc:686] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 1 LEADER]: Becoming Leader. State: Replica: 983e2e7717d24f87bdfb37a18661757e, State: Running, Role: LEADER
I20260810 23:09:03.214638 29980 consensus_queue.cc:237] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.217949 29492 catalog_manager.cc:5495] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e reported cstate change: term changed from 0 to 1, leader changed from <none> to 983e2e7717d24f87bdfb37a18661757e (127.14.75.2). New cstate: current_term: 1 leader_uuid: "983e2e7717d24f87bdfb37a18661757e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } health_report { overall_health: UNKNOWN } } }
I20260810 23:09:03.236141 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.232740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.14.75.2:37179)
0810 23:09:03.232887 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:03.232891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:03.232904 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:09:03.232981 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:03.232983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:03.233039 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:03.233130 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:03.233350 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.233936 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.234768 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:03.234991 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:03.235075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:03.235362 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:03.235808 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:09:03.235905 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:03.235956 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:09:03.236008 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:09:03.242026 29637 raft_consensus.cc:1261] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Refusing update from remote peer 983e2e7717d24f87bdfb37a18661757e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:03.242029 29916 raft_consensus.cc:1261] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Refusing update from remote peer 983e2e7717d24f87bdfb37a18661757e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:03.242712 29980 consensus_queue.cc:1035] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [LEADER]: Connected to new peer: Peer: permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:03.242936 29984 consensus_queue.cc:1035] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [LEADER]: Connected to new peer: Peer: permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:03.258911 29491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:03.259102 29491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:03.261740 29491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Sending DeleteTablet for 3 replicas of tablet 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:03.262571 29616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:03 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:45070
I20260810 23:09:03.262674 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29530
I20260810 23:09:03.262809 29895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:03 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46366
I20260810 23:09:03.263356 29759 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:03 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:34130
I20260810 23:09:03.263960 29999 tablet_replica.cc:330] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: stopping tablet replica
I20260810 23:09:03.264325 29999 raft_consensus.cc:2229] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:09:03.264766 29999 raft_consensus.cc:2258] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:03.265864 29998 tablet_replica.cc:330] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: stopping tablet replica
I20260810 23:09:03.266269 29998 raft_consensus.cc:2229] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:03.266274 29999 ts_tablet_manager.cc:1905] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:03.266686 29998 raft_consensus.cc:2258] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:03.268213 29998 ts_tablet_manager.cc:1905] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:03.269955 29999 ts_tablet_manager.cc:1918] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:03.270061 29999 log.cc:1196] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:03.270433 29999 ts_tablet_manager.cc:1939] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 983e2e7717d24f87bdfb37a18661757e: Deleting consensus metadata
I20260810 23:09:03.271780 29475 catalog_manager.cc:4841] TS 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 (table test-table [id=e94e2eff5d5f452c8eb56056cfa416f2]) successfully deleted
I20260810 23:09:03.272222 29998 ts_tablet_manager.cc:1918] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260810 23:09:03.272372 29477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:33455 (error 108)
I20260810 23:09:03.272434 29998 log.cc:1196] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:03.273319 29998 ts_tablet_manager.cc:1939] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 852bc9ad18454a1b93368bdea2ecdeea: Deleting consensus metadata
I20260810 23:09:03.273350 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29688
I20260810 23:09:03.274115 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.273328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:33455 (local address 127.0.0.1:45086)
0810 23:09:03.273944 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.274018 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524}
W20260810 23:09:03.274312 29477 catalog_manager.cc:4584] TS 365b6146a7a6479f83acb9718a1a3879 (127.14.75.1:33455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f4d8a319ee94dc9b5e7d08012b6fd58: Network error: Client connection negotiation failed: client connection to 127.14.75.1:33455: connect: Connection refused (error 111)
I20260810 23:09:03.276675 29475 catalog_manager.cc:4841] TS 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): tablet 8f4d8a319ee94dc9b5e7d08012b6fd58 (table test-table [id=e94e2eff5d5f452c8eb56056cfa416f2]) successfully deleted
I20260810 23:09:03.280666 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29829
I20260810 23:09:03.287106 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 29456
I20260810 23:09:03.294800 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--webserver_interface=127.14.75.62
--webserver_port=34553
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:03.410229 30001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:03.410583 30001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:03.410667 30001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:03.414942 30001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:09:03.415055 30001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:03.415095 30001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:03.415153 30001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:09:03.415213 30001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:09:03.420854 30001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=34553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:03.422068 30001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:03.423213 30001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:03.432013 30009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.432209 30007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:03.432392 30001 server_base.cc:1034] running on GCE node
W20260810 23:09:03.432441 30006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:03.433013 30001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:03.434238 30001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:03.435438 30001 hybrid_clock.cc:648] HybridClock initialized: now 1786403343435404 us; error 51 us; skew 500 ppm
Aug 10 23:09:03 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:03.438514 30001 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:09:03.439502 30001 webserver.cc:460] Webserver started at http://127.14.75.62:34553/ using document root <none> and password file <none>
I20260810 23:09:03.440085 30001 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:03.440191 30001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:03.444830 30001 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:09:03.447250 30016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.448596 30001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:09:03.448751 30001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "ffda23007075428c90158bfe73ffeccf"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "c7ba2352c78ebf773f9945198840aa03"
server_key_iv: "3ba98025597fa4cf62e152486c1cfc34"
server_key_version: "encryptionkey@0"
I20260810 23:09:03.449534 30001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:03.477480 30001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:03.481029 30001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:03.481268 30001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:03.488919 30001 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42755
I20260810 23:09:03.489277 30071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42755 every 8 connection(s)
I20260810 23:09:03.490132 30001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:09:03.492308 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30001
I20260810 23:09:03.492838 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:33455
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=36493
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:03.495399 30072 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:09:03.497229 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap starting.
I20260810 23:09:03.515487 30072 log.cc:824] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:03.527604 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:03.528098 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: Bootstrap complete.
I20260810 23:09:03.533000 30072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:03.533457 30072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Initialized, Role: FOLLOWER
I20260810 23:09:03.534049 30072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:03.534183 30072 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:09:03.534268 30072 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:09:03.534373 30072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:09:03.536425 30072 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:03.536826 30072 leader_election.cc:302] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffda23007075428c90158bfe73ffeccf; no voters:
I20260810 23:09:03.537285 30072 leader_election.cc:288] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:09:03.537488 30076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:09:03.537909 30076 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [term 3 LEADER]: Becoming Leader. State: Replica: ffda23007075428c90158bfe73ffeccf, State: Running, Role: LEADER
I20260810 23:09:03.538352 30076 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } }
I20260810 23:09:03.538542 30072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:09:03.540117 30078 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffda23007075428c90158bfe73ffeccf. Latest consensus state: current_term: 3 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:03.540122 30077 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ffda23007075428c90158bfe73ffeccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffda23007075428c90158bfe73ffeccf" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42755 } } }
I20260810 23:09:03.540253 30078 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:03.540273 30077 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:03.544291 30091 catalog_manager.cc:1259] Loaded cluster ID: 362d3d85e4f74ab1af0d443de73780ef
I20260810 23:09:03.544365 30091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: loading cluster ID for follower catalog manager: success
W20260810 23:09:03.545509 30091 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ffda23007075428c90158bfe73ffeccf: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260810 23:09:03.545598 30091 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260810 23:09:03.545670 30088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:09:03.549077 30088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:09:03.549600 30088 catalog_manager.cc:1259] Loaded cluster ID: 362d3d85e4f74ab1af0d443de73780ef
I20260810 23:09:03.549672 30088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:09:03.550387 30088 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fedb069d9ad google::LogMessage::Fail() at ??:0
@ 0x7fedb06a088a google::LogMessage::SendToLog() at ??:0
@ 0x7fedb069d4de google::LogMessage::Flush() at ??:0
@ 0x7fedb069ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fedbafdcfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fedbafddd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fedbafda579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fedbb01e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fedb28ef3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fedb28efc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fedb28f13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fedb28e0512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fedb3f246db start_thread at ??:0
@ 0x7fedaf19b61f clone at ??:0
W20260810 23:09:03.636217 30074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:03.636669 30074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:03.636751 30074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:03.641577 30074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:03.641654 30074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:03.641752 30074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:09:03.647595 30074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:33455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=36493
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:03.648947 30074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:03.650151 30074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:03.660391 30116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.660394 30114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.660387 30113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:03.660719 30074 server_base.cc:1034] running on GCE node
I20260810 23:09:03.661201 30074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:03.661965 30074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:03.663174 30074 hybrid_clock.cc:648] HybridClock initialized: now 1786403343663149 us; error 46 us; skew 500 ppm
Aug 10 23:09:03 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:03.666201 30074 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:09:03.667223 30074 webserver.cc:460] Webserver started at http://127.14.75.1:36493/ using document root <none> and password file <none>
I20260810 23:09:03.667855 30074 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:03.667922 30074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:03.672194 30074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:09:03.674846 30124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.676270 30074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:09:03.676424 30074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "365b6146a7a6479f83acb9718a1a3879"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "4a6af9286d415e2acf3456c41dc60ce1"
server_key_iv: "5368dba2f77fd53dd657ce4cbb624675"
server_key_version: "encryptionkey@0"
I20260810 23:09:03.677026 30074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:03.691987 30074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:03.695655 30074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:03.695887 30074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:03.696543 30074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:03.697755 30131 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:09:03.702222 30074 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:09:03.702328 30074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 23:09:03.702420 30074 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:09:03.704316 30074 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:09:03.704406 30074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:09:03.704695 30131 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Bootstrap starting.
I20260810 23:09:03.715535 30074 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:33455
I20260810 23:09:03.715857 30244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:33455 every 8 connection(s)
I20260810 23:09:03.716934 30074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:09:03.720975 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30074
I20260810 23:09:03.721585 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:37179
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=34299
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:03.726707 30131 log.cc:824] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:03.734200 30131 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:03.734633 30131 tablet_bootstrap.cc:492] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Bootstrap complete.
I20260810 23:09:03.735862 30131 ts_tablet_manager.cc:1397] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260810 23:09:03.738772 30131 raft_consensus.cc:348] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.739235 30131 raft_consensus.cc:729] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365b6146a7a6479f83acb9718a1a3879, State: Initialized, Role: FOLLOWER
I20260810 23:09:03.739777 30131 consensus_queue.cc:260] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:03.740806 30131 ts_tablet_manager.cc:1428] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260810 23:09:03.844344 30247 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:09:03.719529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:32901)
0810 23:09:03.720867 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.720934 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:03.722095 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:03.843248 (+121153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":165,"threads_started":1}
W20260810 23:09:03.845008 30245 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:42755 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:42755: Client connection negotiation failed: client connection to 127.14.75.62:42755: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:09:03.845912 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.845601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:51231)
0810 23:09:03.845748 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.845835 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:09:03.846653 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.846404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:52655)
0810 23:09:03.846523 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.846579 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:09:03.846853 30245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:09:03.859215 30248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:03.859501 30248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:03.859548 30248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:03.863890 30248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:03.863969 30248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:03.864049 30248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:09:03.869760 30248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:37179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=34299
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:03.870904 30248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:03.871956 30248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:03.881786 30255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.881798 30257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:03.881786 30259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:03.882135 30248 server_base.cc:1034] running on GCE node
I20260810 23:09:03.882558 30248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:03.883267 30248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:03.884441 30248 hybrid_clock.cc:648] HybridClock initialized: now 1786403343884430 us; error 34 us; skew 500 ppm
Aug 10 23:09:03 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:03.887468 30248 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:09:03.888522 30248 webserver.cc:460] Webserver started at http://127.14.75.2:34299/ using document root <none> and password file <none>
I20260810 23:09:03.889122 30248 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:03.889194 30248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:03.893195 30248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:03.895511 30267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.896823 30248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:09:03.896970 30248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "983e2e7717d24f87bdfb37a18661757e"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "f88cd05066f71a39754095614b348e28"
server_key_iv: "64d097c66da00935bb7fd8045cd73458"
server_key_version: "encryptionkey@0"
I20260810 23:09:03.897621 30248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:03.923161 30248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:03.926543 30248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:03.926785 30248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:03.927438 30248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:03.928319 30248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:03.928395 30248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.928469 30248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:03.928512 30248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:03.938125 30248 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:37179
I20260810 23:09:03.938174 30385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:37179 every 8 connection(s)
I20260810 23:09:03.939179 30248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:09:03.941325 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30248
I20260810 23:09:03.941819 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:40191
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:42755
--webserver_port=34807
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:03.943691 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.941456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:56999)
0810 23:09:03.942029 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.942669 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
W20260810 23:09:03.944312 30386 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:42755 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:42755: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
I20260810 23:09:03.945186 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.944852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:46643)
0810 23:09:03.945010 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.945066 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:09:03.946137 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:03.945823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:51565)
0810 23:09:03.945976 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:03.946055 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 23:09:03.946415 30386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:09:04.066926 30389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:04.067256 30389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:04.067307 30389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:04.072309 30389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:04.072471 30389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:04.072629 30389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:09:04.079242 30389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:40191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=34807
--tserver_master_addrs=127.14.75.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:04.080452 30389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:04.081722 30389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:04.092723 30398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:04.092697 30394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:04.092693 30395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:04.093189 30389 server_base.cc:1034] running on GCE node
I20260810 23:09:04.093710 30389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:04.094424 30389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:04.095606 30389 hybrid_clock.cc:648] HybridClock initialized: now 1786403344095566 us; error 52 us; skew 500 ppm
Aug 10 23:09:04 dist-test-slave-srgv krb5kdc[28891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403344, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:04.098737 30389 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:09:04.099699 30389 webserver.cc:460] Webserver started at http://127.14.75.3:34807/ using document root <none> and password file <none>
I20260810 23:09:04.100361 30389 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:04.100430 30389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:04.104636 30389 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:09:04.107023 30406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:04.108372 30389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:09:04.108530 30389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "852bc9ad18454a1b93368bdea2ecdeea"
format_stamp: "Formatted at 2026-08-10 23:09:01 on dist-test-slave-srgv"
server_key: "eff272cc82a16b44959fd5dc8bb4ce0f"
server_key_iv: "1b1a7fb5828bda6df0d2383257acf3f3"
server_key_version: "encryptionkey@0"
I20260810 23:09:04.109146 30389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:04.133136 30389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:04.136826 30389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:04.137058 30389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:04.137827 30389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:04.138687 30389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:04.138737 30389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:04.138803 30389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:04.138833 30389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:04.147938 30389 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:40191
I20260810 23:09:04.148031 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:40191 every 8 connection(s)
I20260810 23:09:04.149129 30389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:09:04.151286 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30389
I20260810 23:09:04.153740 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:04.151572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:56405)
0810 23:09:04.152148 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:04.152312 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
W20260810 23:09:04.154192 30523 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:42755 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:42755: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
I20260810 23:09:04.154820 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:04.154584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:44281)
0810 23:09:04.154710 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:04.154766 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:09:04.155448 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:04.155260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:54653)
0810 23:09:04.155362 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:04.155404 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:09:04.155637 30523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:09:04.848032 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:04.847537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:45107)
0810 23:09:04.847842 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:04.847910 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260810 23:09:04.947506 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:04.947060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:37025)
0810 23:09:04.947331 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:04.947394 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260810 23:09:05.156849 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.156275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:50191)
0810 23:09:05.156597 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:05.156692 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260810 23:09:05.410766 30530 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:05.410969 30530 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:05.412062 30530 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
Aug 10 23:09:05 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.3@KRBTEST.COM
Aug 10 23:09:05 dist-test-slave-srgv krb5kdc[28891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403343, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.2@KRBTEST.COM
I20260810 23:09:05.428423 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.412540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:48051 (local address 127.14.75.3:40191)
0810 23:09:05.412945 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:05.412972 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:05.413026 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:09:05.414037 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:05.414052 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:05.414488 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:05.414814 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:05.416167 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.417126 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.418412 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.418899 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.419085 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:05.422390 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:05.422400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:05.422419 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:05.422465 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:05.424785 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:05.425630 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:05.425637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:05.425642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:05.425811 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:05.426323 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:05.426327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:05.426328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:05.426949 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:05.427097 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:05.427901 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 23:09:05.428235 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
I20260810 23:09:05.428423 30535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.413169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:56413 (local address 127.14.75.2:37179)
0810 23:09:05.413539 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:05.413559 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:05.413591 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:09:05.414429 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:05.414436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:05.414835 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:05.415112 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:05.416163 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.417122 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.418416 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.418902 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.419080 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:05.422641 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:05.422647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:05.422658 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:05.422693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:05.424700 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:05.425630 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:05.425637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:05.425642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:05.425811 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:05.426323 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:05.426327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:05.426328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:05.426949 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:05.427097 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:05.427901 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 23:09:05.428216 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260810 23:09:05.428431 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.412420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:40191 (local address 127.14.75.1:48051)
0810 23:09:05.412736 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:05.412751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:05.412912 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:05.414913 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:05.414919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:05.415448 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:05.415943 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:05.415959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.417290 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.417293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:05.418277 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:05.418287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.419041 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.419045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:05.419278 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:05.420072 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:05.420089 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:05.422037 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:05.424904 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:05.424907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:05.424908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:05.425474 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:05.425934 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:05.425938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:05.425940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:05.426194 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:05.427127 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:05.427132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:05.427506 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 23:09:05.427736 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 23:09:05.428213 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":302,"spinlock_wait_cycles":23040,"thread_start_us":119,"threads_started":1}
I20260810 23:09:05.428431 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.412897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:37179 (local address 127.14.75.1:56413)
0810 23:09:05.413146 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:05.413157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:05.413236 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:05.415095 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:05.415098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:05.415448 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:05.415943 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:05.415959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.417279 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.417283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:05.418276 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:05.418287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:05.419044 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:05.419048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:05.419281 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:05.420145 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:05.420158 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:05.422398 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:05.424852 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:05.424861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:05.424881 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:05.425474 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:05.425934 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:05.425938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:05.425940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:05.426194 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:05.427122 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:05.427130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:05.427506 (+ 376us) client_negotiation.cc:769] Sending connection context
0810 23:09:05.427736 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 23:09:05.428157 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":224,"spinlock_wait_cycles":29824,"thread_start_us":61,"threads_started":1}
W20260810 23:09:05.431964 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
W20260810 23:09:05.432643 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:05.432753 30126 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879; no voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e
I20260810 23:09:05.432999 30530 raft_consensus.cc:2726] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:09:05.849270 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.848867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:47627)
0810 23:09:05.849103 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:05.849177 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 23:09:05.948930 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:05.948346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:48487)
0810 23:09:05.948732 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:05.948806 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260810 23:09:06.158074 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:06.157633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:50831)
0810 23:09:06.157914 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:06.157977 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260810 23:09:06.850708 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:06.850234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:53397)
0810 23:09:06.850521 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:06.850593 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260810 23:09:06.950330 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:06.949831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:33749)
0810 23:09:06.950134 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:06.950199 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260810 23:09:07.159458 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:07.158976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:49789)
0810 23:09:07.159276 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:07.159341 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260810 23:09:07.304584 30542 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:07.304735 30542 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:07.305241 30542 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
W20260810 23:09:07.306315 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
W20260810 23:09:07.306440 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:07.306514 30126 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879; no voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e
I20260810 23:09:07.306749 30542 raft_consensus.cc:2726] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:09:07.852131 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:07.851631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:40855)
0810 23:09:07.851970 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:07.852028 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260810 23:09:07.951843 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:07.951123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:35559)
0810 23:09:07.951652 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:07.951719 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":71,"threads_started":1}
I20260810 23:09:08.161095 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:08.160607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:55729)
0810 23:09:08.160921 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:08.160985 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260810 23:09:08.853768 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:08.853178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:54863)
0810 23:09:08.853590 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:08.853669 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":100,"threads_started":1}
I20260810 23:09:08.953253 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:08.952784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:59091)
0810 23:09:08.953074 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:08.953139 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260810 23:09:09.162598 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:09.162088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:37881)
0810 23:09:09.162430 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:09.162496 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260810 23:09:09.855229 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:09.854756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:56171)
0810 23:09:09.855061 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:09.855129 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260810 23:09:09.954553 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:09.954109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:41751)
0810 23:09:09.954391 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:09.954453 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260810 23:09:10.163908 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:10.163448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:53271)
0810 23:09:10.163748 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:10.163809 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
I20260810 23:09:10.580968 30553 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:10.581112 30553 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:10.581743 30553 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
W20260810 23:09:10.582662 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
W20260810 23:09:10.582829 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:10.582893 30126 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879; no voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e
I20260810 23:09:10.583050 30553 raft_consensus.cc:2726] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:09:10.856516 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:10.856053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:54477)
0810 23:09:10.856353 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:10.856417 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260810 23:09:10.955925 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:10.955410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:54771)
0810 23:09:10.955752 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:10.955824 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":91,"threads_started":1}
I20260810 23:09:11.165596 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:11.164995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:51383)
0810 23:09:11.165358 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:11.165491 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":103,"threads_started":1}
I20260810 23:09:11.857810 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:11.857320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:39455)
0810 23:09:11.857649 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:11.857708 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":158,"threads_started":1}
I20260810 23:09:11.957140 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:11.956730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:53363)
0810 23:09:11.956986 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:11.957044 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260810 23:09:12.166831 30559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:12.166410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:44427)
0810 23:09:12.166668 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:12.166731 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260810 23:09:12.859198 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:12.858673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:43599)
0810 23:09:12.859002 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:12.859071 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260810 23:09:12.958498 30561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:12.958021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:45699)
0810 23:09:12.958332 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:12.958399 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20260810 23:09:13.168118 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:13.167643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:44175)
0810 23:09:13.167943 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:13.168007 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260810 23:09:13.541550 30563 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:13.541757 30563 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:13.542258 30563 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
W20260810 23:09:13.543064 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
W20260810 23:09:13.543237 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:13.543298 30126 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879; no voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e
I20260810 23:09:13.543454 30563 raft_consensus.cc:2726] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:09:13.860675 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:13.860190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:47895)
0810 23:09:13.860476 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:13.860541 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
I20260810 23:09:13.959928 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:13.959441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:44427)
0810 23:09:13.959726 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:13.959791 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260810 23:09:14.169577 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:14.169044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:38243)
0810 23:09:14.169349 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:14.169447 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":109,"threads_started":1}
I20260810 23:09:14.862140 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:14.861678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:45371)
0810 23:09:14.861968 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:14.862036 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260810 23:09:14.961221 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:14.960772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:49191)
0810 23:09:14.961058 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:14.961123 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260810 23:09:15.171020 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:15.170571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:50407)
0810 23:09:15.170857 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:15.170920 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
I20260810 23:09:15.863533 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:15.863055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:60051)
0810 23:09:15.863372 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:15.863433 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":130,"threads_started":1}
I20260810 23:09:15.962584 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:15.962091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:46591)
0810 23:09:15.962384 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:15.962441 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260810 23:09:16.033983 30573 raft_consensus.cc:482] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:16.034119 30573 raft_consensus.cc:504] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852bc9ad18454a1b93368bdea2ecdeea" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 40191 } } peers { permanent_uuid: "983e2e7717d24f87bdfb37a18661757e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 37179 } } peers { permanent_uuid: "365b6146a7a6479f83acb9718a1a3879" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 33455 } }
I20260810 23:09:16.034578 30573 leader_election.cc:288] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191), 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179)
W20260810 23:09:16.035481 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 852bc9ad18454a1b93368bdea2ecdeea (127.14.75.3:40191): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
W20260810 23:09:16.035609 30126 leader_election.cc:341] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 983e2e7717d24f87bdfb37a18661757e (127.14.75.2:37179): Not found: Tablet not found: 8f4d8a319ee94dc9b5e7d08012b6fd58
I20260810 23:09:16.035670 30126 leader_election.cc:302] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 365b6146a7a6479f83acb9718a1a3879; no voters: 852bc9ad18454a1b93368bdea2ecdeea, 983e2e7717d24f87bdfb37a18661757e
I20260810 23:09:16.035885 30573 raft_consensus.cc:2726] T 8f4d8a319ee94dc9b5e7d08012b6fd58 P 365b6146a7a6479f83acb9718a1a3879 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:09:16.172325 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:16.171899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:55125)
0810 23:09:16.172169 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:16.172228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260810 23:09:16.864813 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:16.864365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:53407)
0810 23:09:16.864638 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:16.864709 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260810 23:09:16.963861 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:16.963371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:41073)
0810 23:09:16.963712 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:16.963767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260810 23:09:17.173550 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:17.173044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:45931)
0810 23:09:17.173301 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:17.173357 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260810 23:09:17.866123 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:17.865689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:60195)
0810 23:09:17.865963 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:17.866024 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260810 23:09:17.965297 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:17.964798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:49115)
0810 23:09:17.965105 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:17.965155 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":91,"threads_started":1}
I20260810 23:09:18.174835 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:18.174367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.3:47669)
0810 23:09:18.174686 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:18.174741 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260810 23:09:18.867586 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:18.867207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.1:42785)
0810 23:09:18.867440 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:18.867491 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260810 23:09:18.966627 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:18.966243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:42755 (local address 127.14.75.2:36371)
0810 23:09:18.966483 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:18.966535 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260810 23:09:19.152326 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30074
I20260810 23:09:19.159418 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30248
I20260810 23:09:19.167795 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30389
2026-08-10T23:09:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19423 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:09:19 dist-test-slave-srgv krb5kdc[30585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:09:19 dist-test-slave-srgv krb5kdc[30585](info): set up 2 sockets
Aug 10 23:09:19 dist-test-slave-srgv krb5kdc[30585](info): commencing operation
krb5kdc: starting...
W20260810 23:09:20.270519 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:09:20 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:09:20.400267 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:20.515950 30599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:20.516306 30599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:20.516383 30599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:20.520627 30599 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:09:20.520744 30599 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:09:20.520771 30599 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:09:20.520792 30599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:09:20.520835 30599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:20.520893 30599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:20.520936 30599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:09:20.520973 30599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:09:20.526690 30599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:20.527895 30599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:20.529001 30599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:20.538033 30607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:20.538048 30604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:20.538141 30599 server_base.cc:1034] running on GCE node
W20260810 23:09:20.538057 30605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:20.538932 30599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:20.540166 30599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:20.541390 30599 hybrid_clock.cc:648] HybridClock initialized: now 1786403360541370 us; error 30 us; skew 500 ppm
Aug 10 23:09:20 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:20.544351 30599 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:09:20.545257 30599 webserver.cc:460] Webserver started at http://127.14.75.62:36263/ using document root <none> and password file <none>
I20260810 23:09:20.545840 30599 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:20.545902 30599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:20.546082 30599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:20.548174 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "c9abf9b219914860b09d02c3e248739e"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "5a78dc5456294564bd1a3ad008d5af49"
server_key_iv: "60d35f291ae295e7acfa0280af6e7fd4"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.548709 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "c9abf9b219914860b09d02c3e248739e"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "5a78dc5456294564bd1a3ad008d5af49"
server_key_iv: "60d35f291ae295e7acfa0280af6e7fd4"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.552527 30599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:09:20.555059 30614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:20.556295 30599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:09:20.556433 30599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "c9abf9b219914860b09d02c3e248739e"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "5a78dc5456294564bd1a3ad008d5af49"
server_key_iv: "60d35f291ae295e7acfa0280af6e7fd4"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.556546 30599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:20.571942 30599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:20.575098 30599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:20.575315 30599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:20.582636 30599 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46689
I20260810 23:09:20.582916 30672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46689 every 8 connection(s)
I20260810 23:09:20.583706 30599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:09:20.586615 30673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:20.587997 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30599
I20260810 23:09:20.588121 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:09:20.588466 14636 external_mini_cluster.cc:1428] Setting key 7052f67e7c036f4e973010fa22ff8563
I20260810 23:09:20.594084 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap starting.
I20260810 23:09:20.596828 30673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:20.597666 30673 log.cc:824] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:09:20 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:20.599993 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: No bootstrap required, opened a new log
I20260810 23:09:20.605175 30673 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:20.605485 30673 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:20.605517 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:20.590201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.14.75.62:46689)
0810 23:09:20.590830 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:20.590855 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:20.590998 (+ 143us) server_negotiation.cc:407] Connection header received
0810 23:09:20.592041 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:20.592066 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:20.592511 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:20.592888 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:20.593690 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.595115 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.596004 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.596700 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.596875 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:20.599698 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:20.599705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:20.599721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:20.599756 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:20.602047 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:20.602557 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:20.602563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:20.602569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:20.602758 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:20.603061 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:20.603066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:20.603069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:20.603633 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:20.603853 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:20.604098 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:09:20.604450 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":122,"threads_started":1}
I20260810 23:09:20.605558 30673 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Initialized, Role: FOLLOWER
I20260810 23:09:20.606048 30673 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:20.606184 30673 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:09:20.606269 30673 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:09:20.606371 30673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:20.607625 30673 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:20.608008 30673 leader_election.cc:302] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9abf9b219914860b09d02c3e248739e; no voters:
I20260810 23:09:20.608386 30673 leader_election.cc:288] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:09:20.608439 30678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:20.608647 30678 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 LEADER]: Becoming Leader. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Running, Role: LEADER
I20260810 23:09:20.608994 30678 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:20.609742 30673 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:09:20.610899 30680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9abf9b219914860b09d02c3e248739e. Latest consensus state: current_term: 1 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:09:20.610929 30679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:09:20.611035 30680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:20.611040 30679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:20.611557 30687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:09:20.614554 30687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:09:20.619988 30687 catalog_manager.cc:1347] Generated new cluster ID: 6be10703fe1642d399b19d7aad0d49c3
I20260810 23:09:20.620074 30687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:09:20.630602 30687 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:09:20.631963 30687 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:09:20.639348 30687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Generated new TSK 0
I20260810 23:09:20.640216 30687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:09:20.672854 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:20.790032 30699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:20.790369 30699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:20.790442 30699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:20.794921 30699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:20.795011 30699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:20.795127 30699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:09:20.800877 30699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:20.802167 30699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:20.803282 30699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:20.813544 30705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:20.813717 30699 server_base.cc:1034] running on GCE node
W20260810 23:09:20.813544 30704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:20.813894 30707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:20.814384 30699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:20.815086 30699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:20.816282 30699 hybrid_clock.cc:648] HybridClock initialized: now 1786403360816267 us; error 30 us; skew 500 ppm
Aug 10 23:09:20 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:20.819205 30699 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:09:20.820149 30699 webserver.cc:460] Webserver started at http://127.14.75.1:46651/ using document root <none> and password file <none>
I20260810 23:09:20.820758 30699 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:20.820854 30699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:20.821102 30699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:20.823355 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "bbb02d18786e4537a4f3bc088873e5ec"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "0039ebd66fb5eefdff24647f54d130ed"
server_key_iv: "1f86b721081d0789eab6b965f3b88720"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.823978 30699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb02d18786e4537a4f3bc088873e5ec"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "0039ebd66fb5eefdff24647f54d130ed"
server_key_iv: "1f86b721081d0789eab6b965f3b88720"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.828158 30699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:09:20.830845 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:20.832273 30699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:09:20.832444 30699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "bbb02d18786e4537a4f3bc088873e5ec"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "0039ebd66fb5eefdff24647f54d130ed"
server_key_iv: "1f86b721081d0789eab6b965f3b88720"
server_key_version: "encryptionkey@0"
I20260810 23:09:20.832587 30699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:20.853276 30699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:20.856527 30699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:20.856751 30699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:20.857393 30699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:20.858314 30699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:20.858384 30699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:20.858455 30699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:20.858496 30699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:20.867587 30699 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:41217
I20260810 23:09:20.867619 30831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:41217 every 8 connection(s)
I20260810 23:09:20.868638 30699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:09:20.870252 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30699
I20260810 23:09:20.870395 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:09:20.870714 14636 external_mini_cluster.cc:1428] Setting key 2a13c1fc459fc4d7d50e4e557efb1ac7
Aug 10 23:09:20 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403360, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:09:20.884253 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:20.870646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:51881 (local address 127.14.75.62:46689)
0810 23:09:20.870785 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:20.870792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:20.872431 (+ 1639us) server_negotiation.cc:407] Connection header received
0810 23:09:20.873923 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:20.873954 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:20.874047 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:20.874145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:20.875515 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.876060 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.877313 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.877628 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.877754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:20.880364 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:20.880367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:20.880369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:20.880396 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:20.881911 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:20.882471 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:20.882475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:20.882477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:20.882565 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:20.882981 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:20.882985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:20.882987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:20.883192 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:20.883338 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:20.883904 (+ 566us) server_negotiation.cc:299] Negotiation successful
0810 23:09:20.884059 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:09:20.885156 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:20.871164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:51881)
0810 23:09:20.872270 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:20.872327 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:20.873645 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:20.874365 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:20.874377 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:20.874851 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:20.875355 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:20.875369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.876196 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.876200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:20.877190 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:20.877200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:20.877801 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:20.877805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:20.877980 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:20.878686 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:20.878702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:20.880183 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:20.882034 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:20.882040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:20.882058 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:20.882347 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:20.882667 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:20.882671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:20.882675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:20.882887 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:20.883302 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:20.883308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:20.883651 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 23:09:20.883862 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 23:09:20.884271 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":682,"threads_started":1}
I20260810 23:09:20.886582 30832 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:20.886868 30832 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:20.887363 30832 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:20.889068 30634 ts_manager.cc:194] Registered new tserver with Master: bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:20.890918 30634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:51881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:09:20.901985 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:21.017266 30837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:21.017606 30837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:21.017689 30837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:21.021880 30837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:21.021960 30837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:21.022063 30837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:09:21.027659 30837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:21.028836 30837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:21.030259 30837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:21.040577 30845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:21.040580 30842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:21.040630 30837 server_base.cc:1034] running on GCE node
W20260810 23:09:21.040580 30843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:21.041334 30837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:21.042071 30837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:21.043272 30837 hybrid_clock.cc:648] HybridClock initialized: now 1786403361043251 us; error 33 us; skew 500 ppm
Aug 10 23:09:21 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:21.046221 30837 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:09:21.047224 30837 webserver.cc:460] Webserver started at http://127.14.75.2:43777/ using document root <none> and password file <none>
I20260810 23:09:21.047814 30837 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:21.047914 30837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:21.048154 30837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:21.050457 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "a56be787ca273a746503938e1e8baf57"
server_key_iv: "a6639991a00c0fa1f29fd87a88bab160"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.051067 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "a56be787ca273a746503938e1e8baf57"
server_key_iv: "a6639991a00c0fa1f29fd87a88bab160"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.055045 30837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:09:21.057799 30854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.059176 30837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:09:21.059331 30837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "a56be787ca273a746503938e1e8baf57"
server_key_iv: "a6639991a00c0fa1f29fd87a88bab160"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.059461 30837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:21.074731 30837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:21.078070 30837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:21.078311 30837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:21.078938 30837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:21.079816 30837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:21.079890 30837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.079962 30837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:21.080004 30837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.088955 30837 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33241
I20260810 23:09:21.088985 30970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33241 every 8 connection(s)
I20260810 23:09:21.089995 30837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:09:21.099462 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30837
I20260810 23:09:21.099591 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:09:21.099896 14636 external_mini_cluster.cc:1428] Setting key 8f41cdade00d105e4f29b9a434a1857d
Aug 10 23:09:21 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:21.104969 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.092016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:52543 (local address 127.14.75.62:46689)
0810 23:09:21.092204 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.092209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.093551 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 23:09:21.094433 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.094436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.094484 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.094563 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:21.095752 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.096281 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.097386 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.097656 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.097740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.100663 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:21.100667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:21.100668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:21.100694 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:21.102492 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.103181 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.103184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.103185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.103285 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.103703 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.103708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.103711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.103972 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:21.104140 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.104653 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.104800 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:09:21.105986 30973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.092411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:52543)
0810 23:09:21.093348 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.093393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.094241 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.094724 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.094731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.095132 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:21.095616 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.095631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.096388 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.096390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.097280 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.097289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.097806 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.097809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.097953 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.098615 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:21.098631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:21.100473 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:21.102613 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:21.102623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:21.102643 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:21.103054 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:21.103392 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:21.103398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:21.103402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:21.103603 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:21.104141 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:21.104146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:21.104396 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.104599 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.104967 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":542,"threads_started":1}
I20260810 23:09:21.107244 30971 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:21.107497 30971 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:21.107992 30971 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:21.109191 30634 ts_manager.cc:194] Registered new tserver with Master: 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241)
I20260810 23:09:21.110069 30634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:52543
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:09:21.130776 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:21.248445 30976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:21.248735 30976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:21.248781 30976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:21.252813 30976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:21.252882 30976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:21.252943 30976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:09:21.258250 30976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:21.259279 30976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:21.260332 30976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:21.270332 30985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:21.270395 30983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:21.270339 30982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:21.270738 30976 server_base.cc:1034] running on GCE node
I20260810 23:09:21.271186 30976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:21.271854 30976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:21.273048 30976 hybrid_clock.cc:648] HybridClock initialized: now 1786403361273025 us; error 34 us; skew 500 ppm
Aug 10 23:09:21 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:21.275904 30976 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:09:21.276835 30976 webserver.cc:460] Webserver started at http://127.14.75.3:42929/ using document root <none> and password file <none>
I20260810 23:09:21.277498 30976 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:21.277590 30976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:21.277829 30976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:09:21.279908 30976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "317fd148172709206ec660e70fb52607"
server_key_iv: "d4b21a7bf5c79b7bc1cbb33419866ebd"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.280500 30976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "317fd148172709206ec660e70fb52607"
server_key_iv: "d4b21a7bf5c79b7bc1cbb33419866ebd"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.284366 30976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 23:09:21.286895 30992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.288159 30976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:09:21.288291 30976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "317fd148172709206ec660e70fb52607"
server_key_iv: "d4b21a7bf5c79b7bc1cbb33419866ebd"
server_key_version: "encryptionkey@0"
I20260810 23:09:21.288409 30976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:21.314535 30976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:21.317689 30976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:21.317924 30976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:21.318570 30976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:21.319425 30976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:21.319506 30976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.319578 30976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:21.319629 30976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:21.329011 30976 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45673
I20260810 23:09:21.329035 31111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45673 every 8 connection(s)
I20260810 23:09:21.330001 30976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:09:21.338589 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 30976
I20260810 23:09:21.338737 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:09:21.339041 14636 external_mini_cluster.cc:1428] Setting key 1b55fb623d0d230a44ec4acd259f0c2d
Aug 10 23:09:21 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403361, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:21.344841 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.331997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:52809 (local address 127.14.75.62:46689)
0810 23:09:21.332171 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.332174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.333240 (+ 1066us) server_negotiation.cc:407] Connection header received
0810 23:09:21.334251 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.334255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.334317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.334414 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:21.335681 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.336220 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.337529 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.337830 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.337943 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.340917 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:21.340921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:21.340924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:21.340952 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:21.342658 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.343224 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.343228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.343230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.343316 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.343694 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.343697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.343699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.343890 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:21.344008 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.344547 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.344673 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:09:21.345826 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.332409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:52809)
0810 23:09:21.333068 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.333110 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.334022 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.334599 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.334606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.335006 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:21.335512 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.335526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.336351 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.336354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.337348 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.337358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.337990 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.337994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.338171 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.338837 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:21.338858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:21.340703 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:21.342798 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:21.342805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:21.342823 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:21.343106 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:21.343419 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:21.343423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:21.343425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:21.343580 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:21.344007 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:21.344015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:21.344300 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.344508 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.344868 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":115,"threads_started":1}
I20260810 23:09:21.347255 31112 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:21.347540 31112 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:21.348074 31112 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:21.349207 30634 ts_manager.cc:194] Registered new tserver with Master: f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
I20260810 23:09:21.349833 30634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:52809
I20260810 23:09:21.353694 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:09:21.363866 30676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.355892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.14.75.62:46689)
0810 23:09:21.356078 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.356083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.356235 (+ 152us) server_negotiation.cc:407] Connection header received
0810 23:09:21.356408 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.356416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.356458 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.356548 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:21.357286 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.357884 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.358741 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.358966 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.359085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.360630 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:21.360634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:21.360637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:21.360667 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:21.362018 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.362466 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.362470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.362472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.362593 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:21.362911 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:21.362914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:21.362916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:21.363150 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:21.363326 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.363573 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.363697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:09:21.367082 30634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:09:21.370282 30634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:09:21.385885 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.380081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.0.0.1:44860)
0810 23:09:21.380979 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.380993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.381085 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.381629 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.381632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.381651 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.381743 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.381749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.383094 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.383098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.384314 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.384327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.385192 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.385196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.385310 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.385357 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.385541 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.385701 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260810 23:09:21.386101 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.380335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.14.75.1:41217)
0810 23:09:21.380715 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.380730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.381197 (+ 467us) server_negotiation.cc:407] Connection header received
0810 23:09:21.381280 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.381285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.381483 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.381627 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.381877 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.382943 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.384447 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.385035 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.385173 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.385328 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.385853 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.385965 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":90,"threads_started":1}
I20260810 23:09:21.385942 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.379375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33241 (local address 127.0.0.1:46498)
0810 23:09:21.380388 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.380402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.380505 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.381376 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.381378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.381394 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.381544 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.381549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.382769 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.382773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.383969 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.383978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.385394 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.385397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.385589 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.385622 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.385706 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.385798 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":47,"thread_start_us":122,"threads_started":1}
I20260810 23:09:21.386667 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.380178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.14.75.2:33241)
0810 23:09:21.380825 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.380838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.380860 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:09:21.380934 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.380941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.381117 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.381282 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.381832 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.382618 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.384146 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.385031 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.385147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.385302 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.386401 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.386517 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":146,"threads_started":1}
I20260810 23:09:21.387888 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.379341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45673 (local address 127.0.0.1:33218)
0810 23:09:21.379873 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.379932 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.380071 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.380819 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.380825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.380850 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.381015 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.381026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.382581 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.382585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.384144 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.384155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.386030 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.386033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.387551 (+ 1518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.387578 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.387666 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.387756 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260810 23:09:21.387985 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.379472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.14.75.3:45673)
0810 23:09:21.380012 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.380025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.380050 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:09:21.380322 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.380327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.380544 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.380705 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.381160 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.382423 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.384282 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.385040 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.385146 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.385302 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.387725 (+ 2423us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.387843 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":196,"threads_started":1}
I20260810 23:09:21.389727 30764 tablet_service.cc:1467] Processing CreateTablet for tablet 77c6fbc4a0dc465e876779412a9251cb (DEFAULT_TABLE table=test-table [id=5170497855234a52b62d653425afd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:21.389751 30904 tablet_service.cc:1467] Processing CreateTablet for tablet 77c6fbc4a0dc465e876779412a9251cb (DEFAULT_TABLE table=test-table [id=5170497855234a52b62d653425afd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:21.390772 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c6fbc4a0dc465e876779412a9251cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:21.390785 30904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c6fbc4a0dc465e876779412a9251cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:21.391307 31042 tablet_service.cc:1467] Processing CreateTablet for tablet 77c6fbc4a0dc465e876779412a9251cb (DEFAULT_TABLE table=test-table [id=5170497855234a52b62d653425afd91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:21.392370 31042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c6fbc4a0dc465e876779412a9251cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:21.397043 31130 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap starting.
I20260810 23:09:21.398617 31131 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Bootstrap starting.
I20260810 23:09:21.399453 31130 tablet_bootstrap.cc:654] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:21.400339 31130 log.cc:824] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:21.400700 31132 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Bootstrap starting.
I20260810 23:09:21.401319 31131 tablet_bootstrap.cc:654] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:21.402089 31131 log.cc:824] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:21.402876 31130 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: No bootstrap required, opened a new log
I20260810 23:09:21.403079 31130 ts_tablet_manager.cc:1397] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:21.403575 31132 tablet_bootstrap.cc:654] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:21.404433 31131 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: No bootstrap required, opened a new log
I20260810 23:09:21.404431 31132 log.cc:824] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:21.404635 31131 ts_tablet_manager.cc:1397] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:21.406490 31132 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: No bootstrap required, opened a new log
I20260810 23:09:21.406682 31132 ts_tablet_manager.cc:1397] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:21.407862 31131 raft_consensus.cc:348] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.407876 31130 raft_consensus.cc:348] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.408128 31130 raft_consensus.cc:374] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:21.408130 31131 raft_consensus.cc:374] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:21.408182 31131 raft_consensus.cc:729] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eafdec34aa443cbbd569d17e9b7c36e, State: Initialized, Role: FOLLOWER
I20260810 23:09:21.408192 31130 raft_consensus.cc:729] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb02d18786e4537a4f3bc088873e5ec, State: Initialized, Role: FOLLOWER
I20260810 23:09:21.408632 31130 consensus_queue.cc:260] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.408632 31131 consensus_queue.cc:260] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.409737 31130 ts_tablet_manager.cc:1428] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 23:09:21.410142 30832 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
I20260810 23:09:21.410226 31131 ts_tablet_manager.cc:1428] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:09:21.410580 30971 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
I20260810 23:09:21.410687 31132 raft_consensus.cc:348] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.410959 31132 raft_consensus.cc:374] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:21.411008 31132 raft_consensus.cc:729] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e13b5b6bbb426789590cd7458bf3e3, State: Initialized, Role: FOLLOWER
I20260810 23:09:21.411461 31132 consensus_queue.cc:260] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.412107 31112 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
I20260810 23:09:21.412444 31132 ts_tablet_manager.cc:1428] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260810 23:09:21.581055 31113 tablet.cc:2369] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:09:21.591168 30972 tablet.cc:2369] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:09:21.620111 30833 tablet.cc:2369] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:09:21.787501 31138 raft_consensus.cc:482] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:21.787753 31138 raft_consensus.cc:504] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.789036 31138 leader_election.cc:288] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:21.793237 31137 raft_consensus.cc:482] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:21.793490 31137 raft_consensus.cc:504] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.794263 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.789548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53693 (local address 127.14.75.2:33241)
0810 23:09:21.789717 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.789721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.789942 (+ 221us) server_negotiation.cc:407] Connection header received
0810 23:09:21.790003 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.790006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.790053 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.790131 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.790397 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.791158 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.793029 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.793626 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.793764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.793833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.794057 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.794137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:09:21.794268 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.789387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33241 (local address 127.14.75.3:53693)
0810 23:09:21.789829 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.789843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.789934 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.790162 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.790166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.790185 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.790286 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.790294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.791283 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.791287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.792831 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.792844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.793790 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.793793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.793899 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.793927 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.794015 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.794098 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":99,"thread_start_us":109,"threads_started":1}
I20260810 23:09:21.794910 31137 leader_election.cc:288] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673), bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:21.795032 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.789363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.14.75.3:60361)
0810 23:09:21.789538 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.789557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.789676 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.790039 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.790041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.790061 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.790215 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.790223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.791283 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.791287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.792831 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.792844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.794610 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.794614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.794733 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.794755 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.794832 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.794907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":145}
I20260810 23:09:21.795113 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.789485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:60361 (local address 127.14.75.1:41217)
0810 23:09:21.789629 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.789633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.789649 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:09:21.789835 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.789838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.789888 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.790003 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.790361 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.791158 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.793865 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.794506 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.794610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.794678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.794906 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.794984 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:09:21.795727 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb02d18786e4537a4f3bc088873e5ec" is_pre_election: true
I20260810 23:09:21.795899 30924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" is_pre_election: true
I20260810 23:09:21.796058 30785 raft_consensus.cc:2454] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e13b5b6bbb426789590cd7458bf3e3 in term 0.
I20260810 23:09:21.796084 30924 raft_consensus.cc:2454] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e13b5b6bbb426789590cd7458bf3e3 in term 0.
I20260810 23:09:21.796576 30993 leader_election.cc:302] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6eafdec34aa443cbbd569d17e9b7c36e, bbb02d18786e4537a4f3bc088873e5ec, f1e13b5b6bbb426789590cd7458bf3e3; no voters:
I20260810 23:09:21.796869 31138 raft_consensus.cc:2781] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:09:21.796952 31138 raft_consensus.cc:482] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:09:21.796993 31138 raft_consensus.cc:3037] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:21.798395 31138 raft_consensus.cc:504] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.798851 31138 leader_election.cc:288] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [CANDIDATE]: Term 1 election: Requested vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:21.799216 30924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
I20260810 23:09:21.799299 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb02d18786e4537a4f3bc088873e5ec"
I20260810 23:09:21.799396 30924 raft_consensus.cc:3037] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:21.799453 30785 raft_consensus.cc:3037] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:21.800482 30924 raft_consensus.cc:2454] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e13b5b6bbb426789590cd7458bf3e3 in term 1.
I20260810 23:09:21.800573 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.795624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:43203 (local address 127.14.75.1:41217)
0810 23:09:21.795735 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.795739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.796554 (+ 815us) server_negotiation.cc:407] Connection header received
0810 23:09:21.796623 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.796629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.796687 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.796762 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.797121 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.797709 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.799319 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.799893 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.799966 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.800024 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.800373 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.800450 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:09:21.800894 30785 raft_consensus.cc:2454] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e13b5b6bbb426789590cd7458bf3e3 in term 1.
I20260810 23:09:21.800936 30993 leader_election.cc:302] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3; no voters:
I20260810 23:09:21.800992 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.795532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.14.75.2:43203)
0810 23:09:21.796438 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.796452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.796546 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.796861 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.796867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.796888 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.796994 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.797001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.797823 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.797826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.799134 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.799144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.800122 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.800125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.800235 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.800259 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.800716 (+ 457us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.800823 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":60,"threads_started":1}
I20260810 23:09:21.801209 31138 raft_consensus.cc:2781] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:21.801520 31138 raft_consensus.cc:686] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 LEADER]: Becoming Leader. State: Replica: f1e13b5b6bbb426789590cd7458bf3e3, State: Running, Role: LEADER
I20260810 23:09:21.801752 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb02d18786e4537a4f3bc088873e5ec" is_pre_election: true
I20260810 23:09:21.801931 30785 raft_consensus.cc:2379] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6eafdec34aa443cbbd569d17e9b7c36e in current term 1: Already voted for candidate f1e13b5b6bbb426789590cd7458bf3e3 in this term.
I20260810 23:09:21.801931 31138 consensus_queue.cc:237] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:21.802456 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.796388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:36329 (local address 127.14.75.3:45673)
0810 23:09:21.796712 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.796718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.796733 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:09:21.796779 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.796782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.796827 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.796894 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:21.797208 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.797894 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.801222 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.801855 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.801957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.802024 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.802281 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.802353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260810 23:09:21.802479 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.795204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45673 (local address 127.14.75.2:36329)
0810 23:09:21.795770 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.795784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:21.795915 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:21.796981 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:21.796984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:21.796996 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:21.797110 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.797116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.798028 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.798032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.799154 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:21.799160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.802024 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.802028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:21.802123 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.802150 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:09:21.802264 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:09:21.802340 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":74,"threads_started":1}
I20260810 23:09:21.802948 31063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c6fbc4a0dc465e876779412a9251cb" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" is_pre_election: true
I20260810 23:09:21.803598 30856 leader_election.cc:302] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6eafdec34aa443cbbd569d17e9b7c36e; no voters: bbb02d18786e4537a4f3bc088873e5ec, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:09:21.804131 31137 raft_consensus.cc:2726] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:09:21.805891 30634 catalog_manager.cc:5495] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20260810 23:09:21.847697 31125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.843824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.14.75.3:45673)
0810 23:09:21.843984 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:21.843991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:21.844011 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:09:21.844189 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:21.844194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:21.844284 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:21.844436 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:21.844745 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.845250 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.846178 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:21.846399 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:21.846512 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:21.846846 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:21.847277 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:09:21.847421 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:21.847462 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 23:09:21.847515 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:09:21.855684 30924 raft_consensus.cc:1261] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Refusing update from remote peer f1e13b5b6bbb426789590cd7458bf3e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:21.855667 30785 raft_consensus.cc:1261] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Refusing update from remote peer f1e13b5b6bbb426789590cd7458bf3e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:21.856477 31138 consensus_queue.cc:1035] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:21.856678 31142 consensus_queue.cc:1035] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:21.868520 31146 mvcc.cc:205] Tried to move back new op lower bound from 7317108170151677952 to 7317108169944154112. Current Snapshot: MvccSnapshot[applied={T|T < 7317108170151677952 or (T in {7317108170151677952})}]
I20260810 23:09:21.874346 30633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:21.874541 30633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:21.876917 30633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Sending DeleteTablet for 3 replicas of tablet 77c6fbc4a0dc465e876779412a9251cb
I20260810 23:09:21.877844 30904 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c6fbc4a0dc465e876779412a9251cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:21 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46498
I20260810 23:09:21.877852 31042 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c6fbc4a0dc465e876779412a9251cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:21 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:33218
I20260810 23:09:21.878592 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30699
I20260810 23:09:21.886442 31156 tablet_replica.cc:330] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: stopping tablet replica
I20260810 23:09:21.886911 31156 raft_consensus.cc:2229] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:21.887285 31156 raft_consensus.cc:2258] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:21.887827 31157 tablet_replica.cc:330] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: stopping tablet replica
I20260810 23:09:21.888121 31157 raft_consensus.cc:2229] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:09:21.888542 31157 raft_consensus.cc:2258] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:21.889545 31156 ts_tablet_manager.cc:1905] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:21.889997 31157 ts_tablet_manager.cc:1905] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:09:21.892539 30617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:41217 (error 108)
I20260810 23:09:21.892795 31157 ts_tablet_manager.cc:1918] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:21.892901 31157 log.cc:1196] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/77c6fbc4a0dc465e876779412a9251cb
I20260810 23:09:21.893244 31157 ts_tablet_manager.cc:1939] T 77c6fbc4a0dc465e876779412a9251cb P f1e13b5b6bbb426789590cd7458bf3e3: Deleting consensus metadata
I20260810 23:09:21.893554 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30837
I20260810 23:09:21.895710 30617 catalog_manager.cc:4841] TS f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): tablet 77c6fbc4a0dc465e876779412a9251cb (table test-table [id=5170497855234a52b62d653425afd91d]) successfully deleted
I20260810 23:09:21.897351 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.896871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.0.0.1:44862)
0810 23:09:21.897156 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.897240 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260810 23:09:21.897596 30617 catalog_manager.cc:4584] TS bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77c6fbc4a0dc465e876779412a9251cb: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41217: connect: Connection refused (error 111)
I20260810 23:09:21.901830 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30976
I20260810 23:09:21.903504 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:21.902143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33241 (local address 127.0.0.1:46512)
0810 23:09:21.903326 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:21.903394 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.2:33241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1069}
I20260810 23:09:21.908898 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 30599
I20260810 23:09:41.916966 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--webserver_interface=127.14.75.62
--webserver_port=36263
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:09:42.042091 31160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:42.042385 31160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:42.042426 31160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:42.046569 31160 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:09:42.046638 31160 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:09:42.046653 31160 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:09:42.046669 31160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:09:42.046687 31160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:42.046705 31160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:42.046720 31160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:09:42.046736 31160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:09:42.052070 31160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=36263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:42.053144 31160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:42.054243 31160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:09:42.068481 31160 server_base.cc:1034] running on GCE node
W20260810 23:09:42.068441 31167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.068426 31166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.068429 31169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:42.069482 31160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:42.070747 31160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:42.071944 31160 hybrid_clock.cc:648] HybridClock initialized: now 1786403382071931 us; error 30 us; skew 500 ppm
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:42.075156 31160 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:09:42.076151 31160 webserver.cc:460] Webserver started at http://127.14.75.62:36263/ using document root <none> and password file <none>
I20260810 23:09:42.076776 31160 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:42.076890 31160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:42.080935 31160 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:09:42.083207 31176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.084527 31160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:09:42.084681 31160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "c9abf9b219914860b09d02c3e248739e"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "5a78dc5456294564bd1a3ad008d5af49"
server_key_iv: "60d35f291ae295e7acfa0280af6e7fd4"
server_key_version: "encryptionkey@0"
I20260810 23:09:42.085317 31160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:42.100348 31160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:42.103691 31160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:42.103929 31160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:42.111650 31160 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46689
I20260810 23:09:42.111945 31230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46689 every 8 connection(s)
I20260810 23:09:42.112849 31160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:09:42.115355 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31160
I20260810 23:09:42.115864 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:41217
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=46651
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:42.118105 31231 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:09:42.119846 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap starting.
I20260810 23:09:42.137521 31231 log.cc:824] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:42.146050 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:42.146466 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap complete.
I20260810 23:09:42.150548 31231 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:42.150982 31231 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Initialized, Role: FOLLOWER
I20260810 23:09:42.151459 31231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:42.151623 31231 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:09:42.151711 31231 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:09:42.151830 31231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:09:42.153836 31231 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:42.154181 31231 leader_election.cc:302] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9abf9b219914860b09d02c3e248739e; no voters:
I20260810 23:09:42.154582 31231 leader_election.cc:288] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:09:42.154680 31235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:09:42.154882 31235 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 LEADER]: Becoming Leader. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Running, Role: LEADER
I20260810 23:09:42.155258 31235 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:09:42.155917 31231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:09:42.156314 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9abf9b219914860b09d02c3e248739e. Latest consensus state: current_term: 2 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:09:42.156430 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:42.156828 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:09:42.156999 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:09:42.157050 31240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:09:42.162024 31240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:09:42.162865 31240 catalog_manager.cc:1259] Loaded cluster ID: 6be10703fe1642d399b19d7aad0d49c3
I20260810 23:09:42.162948 31240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:09:42.164722 31240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:09:42.165349 31240 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Loaded TSK: 0
I20260810 23:09:42.179741 31240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Generated new TSK 1
I20260810 23:09:42.180037 31240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:09:42.242743 31233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:42.243047 31233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:42.243127 31233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:42.247287 31233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:42.247382 31233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:42.247478 31233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:09:42.253619 31233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=46651
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:42.254796 31233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:42.255913 31233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:42.265857 31259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.265864 31260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.266119 31262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:42.266755 31233 server_base.cc:1034] running on GCE node
I20260810 23:09:42.267252 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:42.268070 31233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:42.269284 31233 hybrid_clock.cc:648] HybridClock initialized: now 1786403382269260 us; error 32 us; skew 500 ppm
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:42.272863 31233 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:09:42.274029 31233 webserver.cc:460] Webserver started at http://127.14.75.1:46651/ using document root <none> and password file <none>
I20260810 23:09:42.274621 31233 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:42.274734 31233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:42.278935 31233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:09:42.281240 31269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.282649 31233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:09:42.282784 31233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "bbb02d18786e4537a4f3bc088873e5ec"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "0039ebd66fb5eefdff24647f54d130ed"
server_key_iv: "1f86b721081d0789eab6b965f3b88720"
server_key_version: "encryptionkey@0"
I20260810 23:09:42.283376 31233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:42.305173 31233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:42.308949 31233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:42.309114 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:42.309803 31233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:42.310897 31276 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:09:42.314991 31233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:09:42.315058 31233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260810 23:09:42.315131 31233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:09:42.316771 31233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:09:42.316833 31233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:09:42.317085 31276 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap starting.
I20260810 23:09:42.327487 31233 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:41217
I20260810 23:09:42.327848 31387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:41217 every 8 connection(s)
I20260810 23:09:42.328549 31233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:09:42.333837 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31233
I20260810 23:09:42.334321 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:33241
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=43777
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:09:42.339123 31276 log.cc:824] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:42.347913 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.330876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:44159 (local address 127.14.75.62:46689)
0810 23:09:42.333147 (+ 2271us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.333166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.333229 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:09:42.333590 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.333610 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.334137 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.336014 (+ 1877us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.336368 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.337725 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.339384 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.339955 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.340153 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.342808 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.342813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.342826 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.342853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.344628 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.345191 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.345196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.345200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.345315 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.345704 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.345708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.345710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.346055 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.346227 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.346757 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.347017 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2109,"thread_start_us":126,"threads_started":1}
I20260810 23:09:42.347924 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.330706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:44159)
0810 23:09:42.331308 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.331368 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.332615 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.334618 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.334627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.335198 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:42.335973 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.335994 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.337900 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.337903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.339231 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.339243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.340139 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.340143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.340344 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.340982 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:42.340998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:42.342587 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:42.344764 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.344770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.344782 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.345048 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.345438 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.345444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.345449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.345594 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.346198 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:42.346205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:42.346458 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.346653 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.347008 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":169,"threads_started":1}
I20260810 23:09:42.349114 31276 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:09:42.349696 31276 tablet_bootstrap.cc:492] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap complete.
I20260810 23:09:42.350543 31388 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:42.350796 31388 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:42.351246 31276 ts_tablet_manager.cc:1397] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.008s
I20260810 23:09:42.351285 31388 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:42.354148 31193 ts_manager.cc:194] Registered new tserver with Master: bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:42.354264 31276 raft_consensus.cc:348] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.354802 31276 raft_consensus.cc:729] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb02d18786e4537a4f3bc088873e5ec, State: Initialized, Role: FOLLOWER
I20260810 23:09:42.355300 31276 consensus_queue.cc:260] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.356369 31276 ts_tablet_manager.cc:1428] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:09:42.359082 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:44159
I20260810 23:09:42.362150 31388 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.1@KRBTEST.COM
I20260810 23:09:42.368779 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.358313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.14.75.1:41217)
0810 23:09:42.359038 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.359052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.359078 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:09:42.359146 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.359150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.359344 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.359481 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.360225 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.360941 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.361913 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.362312 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.362438 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.365205 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.365214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.365223 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.365245 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.366903 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.367463 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.367467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.367469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.367546 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.367826 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.367829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.367830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.368050 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.368158 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.368518 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.368655 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":117,"threads_started":1}
I20260810 23:09:42.368822 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.358247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.0.0.1:46390)
0810 23:09:42.358678 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.358728 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.358855 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.359506 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.359512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.359979 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:42.360112 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.360122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.361067 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.361072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.361787 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.361797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.362444 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.362447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.362558 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.363187 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:42.363201 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:42.365056 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:42.367018 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.367028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.367034 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.367346 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.367635 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.367638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.367640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.367736 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.368162 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:42.368167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:42.368299 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.368447 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.368656 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260810 23:09:42.369928 31320 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c6fbc4a0dc465e876779412a9251cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:21 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46390
I20260810 23:09:42.370936 31398 tablet_replica.cc:330] stopping tablet replica
I20260810 23:09:42.371244 31398 raft_consensus.cc:2229] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:42.371496 31398 raft_consensus.cc:2258] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:42.372448 31398 ts_tablet_manager.cc:1905] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:42.376001 31398 ts_tablet_manager.cc:1918] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:42.376087 31398 log.cc:1196] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal/wals/77c6fbc4a0dc465e876779412a9251cb
I20260810 23:09:42.376367 31398 ts_tablet_manager.cc:1939] T 77c6fbc4a0dc465e876779412a9251cb P bbb02d18786e4537a4f3bc088873e5ec: Deleting consensus metadata
I20260810 23:09:42.378120 31178 catalog_manager.cc:4841] TS bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217): tablet 77c6fbc4a0dc465e876779412a9251cb (table test-table [id=5170497855234a52b62d653425afd91d]) successfully deleted
W20260810 23:09:42.457259 31392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:42.457572 31392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:42.457621 31392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:42.462193 31392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:42.462275 31392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:42.462393 31392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:09:42.468338 31392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:33241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=43777
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:42.469450 31392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:42.470492 31392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:09:42.480293 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.480266 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.480275 31406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:42.480680 31392 server_base.cc:1034] running on GCE node
I20260810 23:09:42.481141 31392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:42.481865 31392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:42.483042 31392 hybrid_clock.cc:648] HybridClock initialized: now 1786403382483023 us; error 29 us; skew 500 ppm
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:42.485958 31392 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:09:42.486872 31392 webserver.cc:460] Webserver started at http://127.14.75.2:43777/ using document root <none> and password file <none>
I20260810 23:09:42.487392 31392 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:42.487507 31392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:42.491611 31392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:09:42.493806 31413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.495072 31392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:09:42.495227 31392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "a56be787ca273a746503938e1e8baf57"
server_key_iv: "a6639991a00c0fa1f29fd87a88bab160"
server_key_version: "encryptionkey@0"
I20260810 23:09:42.495785 31392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:42.507591 31392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:42.510617 31392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:42.510843 31392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:42.511452 31392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:42.512514 31420 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260810 23:09:42.516228 31420 ts_tablet_manager.cc:1852] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 23:09:42.516623 31420 ts_tablet_manager.cc:1905] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:42.520406 31420 ts_tablet_manager.cc:1918] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:42.520509 31420 log.cc:1196] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/77c6fbc4a0dc465e876779412a9251cb
I20260810 23:09:42.520797 31420 ts_tablet_manager.cc:1939] T 77c6fbc4a0dc465e876779412a9251cb P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting consensus metadata
I20260810 23:09:42.521093 31392 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 23:09:42.521191 31392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260810 23:09:42.521281 31392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:42.521325 31392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.530476 31392 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33241
I20260810 23:09:42.530512 31533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33241 every 8 connection(s)
I20260810 23:09:42.531368 31392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:09:42.532524 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31392
I20260810 23:09:42.533002 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:45673
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=42929
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:42.546284 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.533672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:40941 (local address 127.14.75.62:46689)
0810 23:09:42.533840 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.533845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.534494 (+ 649us) server_negotiation.cc:407] Connection header received
0810 23:09:42.535531 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.535538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.535622 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.535744 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.536884 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.537489 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.538714 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.538949 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.539056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.542151 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.542154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.542156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.542192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.543848 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.544549 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.544552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.544553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.544636 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.545055 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.545057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.545058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.545252 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.545333 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.546041 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.546156 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:09:42.547353 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.533849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:40941)
0810 23:09:42.534338 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.534389 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.535272 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.535891 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.535897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.536276 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:42.536748 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.536761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.537615 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.537619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.538580 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.538590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.539109 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.539113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.539278 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.539994 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:42.540011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:42.541969 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:42.543979 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.543988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.544006 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.544417 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.544738 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.544743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.544747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.544946 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.545368 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:42.545376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:42.545758 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.545997 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.546326 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
I20260810 23:09:42.548508 31534 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:42.548749 31534 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:42.549229 31534 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:42.550273 31193 ts_manager.cc:194] Registered new tserver with Master: 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241)
I20260810 23:09:42.550835 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:40941
W20260810 23:09:42.650861 31537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:09:42.651188 31537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:09:42.651265 31537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:09:42.655418 31537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:09:42.655514 31537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:09:42.655606 31537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:09:42.661399 31537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:45673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=42929
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:09:42.662593 31537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:09:42.663645 31537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:09:42.674387 31537 server_base.cc:1034] running on GCE node
W20260810 23:09:42.674322 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.674322 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:09:42.674328 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:09:42.675110 31537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:09:42.675835 31537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:09:42.677022 31537 hybrid_clock.cc:648] HybridClock initialized: now 1786403382676995 us; error 56 us; skew 500 ppm
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:09:42.679955 31537 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:09:42.680976 31537 webserver.cc:460] Webserver started at http://127.14.75.3:42929/ using document root <none> and password file <none>
I20260810 23:09:42.681615 31537 fs_manager.cc:359] Metadata directory not provided
I20260810 23:09:42.681699 31537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:09:42.685578 31537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:09:42.687625 31554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.689062 31537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:09:42.689163 31537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "317fd148172709206ec660e70fb52607"
server_key_iv: "d4b21a7bf5c79b7bc1cbb33419866ebd"
server_key_version: "encryptionkey@0"
I20260810 23:09:42.689827 31537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:09:42.711099 31537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:09:42.714195 31537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:09:42.714367 31537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:09:42.714931 31537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:09:42.715739 31537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:09:42.715785 31537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.715824 31537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:09:42.715845 31537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:09:42.724890 31537 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45673
I20260810 23:09:42.724912 31672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45673 every 8 connection(s)
I20260810 23:09:42.726059 31537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:09:42.731703 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31537
Aug 10 23:09:42 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403382, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:09:42.740820 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.732472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.14.75.62:46689)
0810 23:09:42.732728 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.732732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.732850 (+ 118us) server_negotiation.cc:407] Connection header received
0810 23:09:42.732952 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.732955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.733005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.733101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.733923 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.734445 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.735316 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.735584 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.735697 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.737239 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.737241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.737243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.737302 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.739041 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.739532 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.739537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.739539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.739617 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.739913 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.739917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.739919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.740097 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.740321 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.740503 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.740629 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 23:09:42.740979 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.727944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:57469 (local address 127.14.75.62:46689)
0810 23:09:42.728085 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.728089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.729142 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 23:09:42.730077 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.730080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.730128 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.730212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.731369 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.732141 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.733755 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.733982 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.734093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.736858 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.736863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.736866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.736893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.738761 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.739291 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.739295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.739296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.739380 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.739756 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.739761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.739762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.739970 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.740081 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.740603 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.740732 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":7552}
I20260810 23:09:42.742154 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.728334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:57469)
0810 23:09:42.729010 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.729050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.729892 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.730375 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.730381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.730755 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:09:42.731240 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.731253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.732281 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.732285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.733624 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.733635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.734148 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.734153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.734317 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.734951 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:09:42.734969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:09:42.736675 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:09:42.738876 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.738882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.738895 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.739179 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.739477 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:09:42.739479 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:09:42.739480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:09:42.739618 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:09:42.740080 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:09:42.740089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:09:42.740359 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.740561 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.741066 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":142,"threads_started":1}
I20260810 23:09:42.743240 31673 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46689
I20260810 23:09:42.743537 31673 heartbeater.cc:464] Registering TS with master...
I20260810 23:09:42.744046 31673 heartbeater.cc:511] Master 127.14.75.62:46689 requested a full tablet report, sending...
I20260810 23:09:42.745005 31193 ts_manager.cc:194] Registered new tserver with Master: f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
I20260810 23:09:42.745646 31193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:57469
I20260810 23:09:42.754774 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:09:42.759440 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.755411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.14.75.62:46689)
0810 23:09:42.755571 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.755575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.755732 (+ 157us) server_negotiation.cc:407] Connection header received
0810 23:09:42.755875 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.755877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.755980 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.756091 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:42.756466 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.757049 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.758156 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.758400 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.758520 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.758856 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:42.759004 (+ 148us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 23:09:42.759294 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50796: token expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:09:42.759512 31677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50796: token expired
W20260810 23:09:42.759840 31116 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.75.62:46689)
I20260810 23:09:42.767027 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.760235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.14.75.62:46689)
0810 23:09:42.760358 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.760361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.760382 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:09:42.760461 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.760463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.760499 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.760561 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:09:42.761146 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.761712 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.762470 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.762674 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.762745 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.764035 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:09:42.764038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:09:42.764040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:09:42.764063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:09:42.765697 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.766055 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.766058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.766059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.766135 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:09:42.766368 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:09:42.766370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:09:42.766371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:09:42.766537 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:09:42.766647 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.766813 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.766909 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:09:42.769512 31193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:09:42.772698 31193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:09:42.780954 31320 tablet_service.cc:1467] Processing CreateTablet for tablet 13f224a603214116b35e6f03029315b9 (DEFAULT_TABLE table=test-table [id=c1143a5ae61f4389898855808d069368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:42.781674 31320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f224a603214116b35e6f03029315b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:42.784636 31276 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap starting.
I20260810 23:09:42.785599 31276 tablet_bootstrap.cc:654] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:42.785665 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.779492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33241 (local address 127.0.0.1:35692)
0810 23:09:42.779617 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.779632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.779725 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.780793 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.780799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.780824 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:42.780942 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.780949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.782584 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.782588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.783800 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.783811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.785076 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.785080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.785202 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.785251 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.785337 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.785451 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":10}
I20260810 23:09:42.785686 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.779855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.14.75.2:33241)
0810 23:09:42.780201 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.780213 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.780243 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:09:42.780335 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.780346 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.780585 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.781112 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:42.781299 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.782456 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.783939 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.784956 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.785060 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.785183 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.785390 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.785510 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":133,"threads_started":1}
I20260810 23:09:42.785996 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.780442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.14.75.3:45673)
0810 23:09:42.781012 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.781026 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.781051 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:09:42.781121 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.781125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.781512 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.781674 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:42.782078 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.782964 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.783987 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.784816 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.784921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.785081 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.785744 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.785847 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":173,"threads_started":1}
I20260810 23:09:42.785674 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.779987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45673 (local address 127.0.0.1:35058)
0810 23:09:42.780236 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.780250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.780334 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.781851 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.781854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.781871 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:42.781951 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.781956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.783078 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.783082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.783824 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.783834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.785162 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.785165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.785272 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.785296 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.785394 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.785502 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260810 23:09:42.787510 31276 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: No bootstrap required, opened a new log
I20260810 23:09:42.787626 31276 ts_tablet_manager.cc:1397] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:09:42.787520 31463 tablet_service.cc:1467] Processing CreateTablet for tablet 13f224a603214116b35e6f03029315b9 (DEFAULT_TABLE table=test-table [id=c1143a5ae61f4389898855808d069368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:42.788142 31276 raft_consensus.cc:348] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.788276 31276 raft_consensus.cc:374] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:42.788323 31276 raft_consensus.cc:729] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb02d18786e4537a4f3bc088873e5ec, State: Initialized, Role: FOLLOWER
I20260810 23:09:42.788446 31276 consensus_queue.cc:260] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.788645 31463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f224a603214116b35e6f03029315b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:42.788739 31276 ts_tablet_manager.cc:1428] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:09:42.789268 31602 tablet_service.cc:1467] Processing CreateTablet for tablet 13f224a603214116b35e6f03029315b9 (DEFAULT_TABLE table=test-table [id=c1143a5ae61f4389898855808d069368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:09:42.790335 31602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f224a603214116b35e6f03029315b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:09:42.793871 31420 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Bootstrap starting.
I20260810 23:09:42.795815 31420 tablet_bootstrap.cc:654] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:42.796538 31420 log.cc:824] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:42.798507 31420 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: No bootstrap required, opened a new log
I20260810 23:09:42.798575 31687 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Bootstrap starting.
I20260810 23:09:42.798710 31420 ts_tablet_manager.cc:1397] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:09:42.800571 31687 tablet_bootstrap.cc:654] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:09:42.801733 31687 log.cc:824] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:09:42.802057 31420 raft_consensus.cc:348] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.802362 31420 raft_consensus.cc:374] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:42.802421 31420 raft_consensus.cc:729] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eafdec34aa443cbbd569d17e9b7c36e, State: Initialized, Role: FOLLOWER
I20260810 23:09:42.802893 31420 consensus_queue.cc:260] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.803591 31534 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
I20260810 23:09:42.803859 31687 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: No bootstrap required, opened a new log
I20260810 23:09:42.803870 31420 ts_tablet_manager.cc:1428] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:09:42.804080 31687 ts_tablet_manager.cc:1397] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:09:42.807346 31687 raft_consensus.cc:348] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.807665 31687 raft_consensus.cc:374] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:09:42.807753 31687 raft_consensus.cc:729] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e13b5b6bbb426789590cd7458bf3e3, State: Initialized, Role: FOLLOWER
I20260810 23:09:42.808266 31687 consensus_queue.cc:260] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.808951 31673 heartbeater.cc:503] Master 127.14.75.62:46689 was elected leader, sending a full tablet report...
I20260810 23:09:42.809209 31687 ts_tablet_manager.cc:1428] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260810 23:09:42.829757 31389 tablet.cc:2369] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:09:42.884264 31689 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:09:42.884465 31689 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.885614 31689 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673), bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:42.889701 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.885895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45673 (local address 127.14.75.2:48083)
0810 23:09:42.886029 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.886048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.886148 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.886422 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.886428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.886453 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:42.886583 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.886590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.887550 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.887552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.888580 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.888591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.889259 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.889265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.889392 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.889446 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.889516 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.889576 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260810 23:09:42.889778 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.885984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:48083 (local address 127.14.75.3:45673)
0810 23:09:42.886110 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.886114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.886128 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:09:42.886253 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.886256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.886316 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.886405 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:42.886691 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.887442 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.888704 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.889113 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.889227 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.889343 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.889555 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.889650 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:09:42.890545 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.886353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:57873 (local address 127.14.75.1:41217)
0810 23:09:42.886689 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.886693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.886799 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:09:42.886906 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.886909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.886973 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.887089 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:09:42.887375 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.888020 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.889313 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.890029 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.890143 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.890312 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.890390 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.890446 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
I20260810 23:09:42.890478 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f224a603214116b35e6f03029315b9" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" is_pre_election: true
I20260810 23:09:42.890583 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.886223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.14.75.2:57873)
0810 23:09:42.886709 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.886722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:09:42.886803 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:09:42.887115 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:09:42.887121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:09:42.887138 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:09:42.887268 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.887274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.888131 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.888135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.889163 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:09:42.889172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.890174 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.890178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:09:42.890272 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.890293 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:09:42.890379 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:09:42.890442 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":190,"threads_started":1}
I20260810 23:09:42.890848 31623 raft_consensus.cc:2454] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eafdec34aa443cbbd569d17e9b7c36e in term 0.
I20260810 23:09:42.891415 31415 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3; no voters:
I20260810 23:09:42.891579 31341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f224a603214116b35e6f03029315b9" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb02d18786e4537a4f3bc088873e5ec" is_pre_election: true
I20260810 23:09:42.891724 31689 raft_consensus.cc:2781] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:09:42.891829 31689 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:09:42.891882 31689 raft_consensus.cc:3037] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:42.891899 31341 raft_consensus.cc:2454] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eafdec34aa443cbbd569d17e9b7c36e in term 0.
I20260810 23:09:42.893080 31689 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.893589 31689 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 election: Requested vote from peers f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673), bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217)
I20260810 23:09:42.894001 31623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f224a603214116b35e6f03029315b9" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
I20260810 23:09:42.894016 31341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f224a603214116b35e6f03029315b9" candidate_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb02d18786e4537a4f3bc088873e5ec"
I20260810 23:09:42.894155 31623 raft_consensus.cc:3037] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:42.894169 31341 raft_consensus.cc:3037] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:09:42.895251 31341 raft_consensus.cc:2454] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eafdec34aa443cbbd569d17e9b7c36e in term 1.
I20260810 23:09:42.895263 31623 raft_consensus.cc:2454] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eafdec34aa443cbbd569d17e9b7c36e in term 1.
I20260810 23:09:42.895531 31415 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eafdec34aa443cbbd569d17e9b7c36e, bbb02d18786e4537a4f3bc088873e5ec; no voters:
I20260810 23:09:42.895738 31689 raft_consensus.cc:2781] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:09:42.896011 31689 raft_consensus.cc:686] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 LEADER]: Becoming Leader. State: Replica: 6eafdec34aa443cbbd569d17e9b7c36e, State: Running, Role: LEADER
I20260810 23:09:42.896369 31689 consensus_queue.cc:237] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:09:42.899650 31192 catalog_manager.cc:5495] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2). New cstate: current_term: 1 leader_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20260810 23:09:42.914372 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.911073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.14.75.2:33241)
0810 23:09:42.911210 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:09:42.911214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:09:42.911229 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:09:42.911328 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:09:42.911330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:09:42.911386 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:09:42.911475 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:09:42.911683 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.912179 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.912929 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:09:42.913159 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:09:42.913235 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:09:42.913554 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:09:42.913965 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:09:42.914056 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:09:42.914115 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:09:42.914185 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:09:42.920198 31341 raft_consensus.cc:1261] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Refusing update from remote peer 6eafdec34aa443cbbd569d17e9b7c36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:42.920208 31623 raft_consensus.cc:1261] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Refusing update from remote peer 6eafdec34aa443cbbd569d17e9b7c36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:09:42.920924 31693 consensus_queue.cc:1035] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:42.921116 31689 consensus_queue.cc:1035] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:09:42.927266 31696 mvcc.cc:205] Tried to move back new op lower bound from 7317108256433811456 to 7317108256346128384. Current Snapshot: MvccSnapshot[applied={T|T < 7317108256433811456}]
I20260810 23:09:42.927585 31697 mvcc.cc:205] Tried to move back new op lower bound from 7317108256433811456 to 7317108256346128384. Current Snapshot: MvccSnapshot[applied={T|T < 7317108256433811456}]
I20260810 23:09:42.929085 31698 mvcc.cc:205] Tried to move back new op lower bound from 7317108256433811456 to 7317108256346128384. Current Snapshot: MvccSnapshot[applied={T|T < 7317108256433811456}]
I20260810 23:09:42.935827 31192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:42.935974 31192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:09:42.938377 31192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Sending DeleteTablet for 3 replicas of tablet 13f224a603214116b35e6f03029315b9
I20260810 23:09:42.939110 31320 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f224a603214116b35e6f03029315b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:42 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:46390
I20260810 23:09:42.939072 31463 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f224a603214116b35e6f03029315b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:42 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:35692
I20260810 23:09:42.939363 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31233
I20260810 23:09:42.939504 31705 tablet_replica.cc:330] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: stopping tablet replica
I20260810 23:09:42.939652 31705 raft_consensus.cc:2229] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:42.939620 31602 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f224a603214116b35e6f03029315b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:09:42 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:35058
I20260810 23:09:42.939785 31705 raft_consensus.cc:2258] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:42.940348 31707 tablet_replica.cc:330] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: stopping tablet replica
I20260810 23:09:42.940747 31707 raft_consensus.cc:2229] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:09:42.941008 31706 tablet_replica.cc:330] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: stopping tablet replica
I20260810 23:09:42.941133 31707 raft_consensus.cc:2258] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:42.941440 31706 raft_consensus.cc:2229] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:09:42.942433 31706 raft_consensus.cc:2258] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:09:42.942716 31707 ts_tablet_manager.cc:1905] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:42.944742 31706 ts_tablet_manager.cc:1905] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:09:42.946476 31707 ts_tablet_manager.cc:1918] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:42.946592 31707 log.cc:1196] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/13f224a603214116b35e6f03029315b9
I20260810 23:09:42.946990 31707 ts_tablet_manager.cc:1939] T 13f224a603214116b35e6f03029315b9 P f1e13b5b6bbb426789590cd7458bf3e3: Deleting consensus metadata
I20260810 23:09:42.947724 31706 ts_tablet_manager.cc:1918] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:09:42.947875 31706 log.cc:1196] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/13f224a603214116b35e6f03029315b9
I20260810 23:09:42.948218 31706 ts_tablet_manager.cc:1939] T 13f224a603214116b35e6f03029315b9 P 6eafdec34aa443cbbd569d17e9b7c36e: Deleting consensus metadata
I20260810 23:09:42.948438 31178 catalog_manager.cc:4841] TS f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): tablet 13f224a603214116b35e6f03029315b9 (table test-table [id=c1143a5ae61f4389898855808d069368]) successfully deleted
W20260810 23:09:42.949535 31178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:41217 (error 108)
I20260810 23:09:42.949608 31177 catalog_manager.cc:4841] TS 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): tablet 13f224a603214116b35e6f03029315b9 (table test-table [id=c1143a5ae61f4389898855808d069368]) successfully deleted
I20260810 23:09:42.949955 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31392
I20260810 23:09:42.950831 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:09:42.950432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:41217 (local address 127.0.0.1:46404)
0810 23:09:42.950639 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 23:09:42.950728 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260810 23:09:42.951004 31178 catalog_manager.cc:4584] TS bbb02d18786e4537a4f3bc088873e5ec (127.14.75.1:41217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13f224a603214116b35e6f03029315b9: Network error: Client connection negotiation failed: client connection to 127.14.75.1:41217: connect: Connection refused (error 111)
I20260810 23:09:42.957558 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31537
I20260810 23:09:42.964205 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31160
I20260810 23:10:02.972157 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--webserver_interface=127.14.75.62
--webserver_port=36263
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:03.090301 31709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:03.090642 31709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:03.090734 31709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:03.094892 31709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:10:03.094980 31709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:10:03.095005 31709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:10:03.095043 31709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:03.095078 31709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:03.095132 31709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:03.095170 31709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:03.095209 31709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:03.100814 31709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=36263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:03.102061 31709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:03.103111 31709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:03.111909 31716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.111914 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.111935 31715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:03.112226 31709 server_base.cc:1034] running on GCE node
I20260810 23:10:03.112982 31709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:03.114235 31709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:03.115401 31709 hybrid_clock.cc:648] HybridClock initialized: now 1786403403115365 us; error 49 us; skew 500 ppm
Aug 10 23:10:03 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:03.118317 31709 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:03.119200 31709 webserver.cc:460] Webserver started at http://127.14.75.62:36263/ using document root <none> and password file <none>
I20260810 23:10:03.119742 31709 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:03.119808 31709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:03.123785 31709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:10:03.126078 31726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.127351 31709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:03.127494 31709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "c9abf9b219914860b09d02c3e248739e"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "5a78dc5456294564bd1a3ad008d5af49"
server_key_iv: "60d35f291ae295e7acfa0280af6e7fd4"
server_key_version: "encryptionkey@0"
I20260810 23:10:03.128057 31709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:03.148480 31709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:03.151751 31709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:03.151959 31709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:03.159303 31709 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46689
I20260810 23:10:03.159586 31780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46689 every 8 connection(s)
I20260810 23:10:03.160432 31709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:03.164777 31782 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:10:03.166797 31782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap starting.
I20260810 23:10:03.170164 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31709
I20260810 23:10:03.170600 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:41217
--local_ip_for_outbound_sockets=127.14.75.1
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=46651
--webserver_interface=127.14.75.1
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:10:03.184952 31782 log.cc:824] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:03.196604 31782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:10:03.197049 31782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Bootstrap complete.
I20260810 23:10:03.201145 31782 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:10:03.201686 31782 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Initialized, Role: FOLLOWER
I20260810 23:10:03.202240 31782 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:10:03.202389 31782 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:03.202482 31782 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:03.202589 31782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:10:03.204550 31782 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:10:03.204912 31782 leader_election.cc:302] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9abf9b219914860b09d02c3e248739e; no voters:
I20260810 23:10:03.205310 31782 leader_election.cc:288] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:10:03.205432 31786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:10:03.205674 31786 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [term 3 LEADER]: Becoming Leader. State: Replica: c9abf9b219914860b09d02c3e248739e, State: Running, Role: LEADER
I20260810 23:10:03.206035 31786 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } }
I20260810 23:10:03.206694 31782 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:03.207578 31788 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:10:03.207636 31787 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9abf9b219914860b09d02c3e248739e. Latest consensus state: current_term: 3 leader_uuid: "c9abf9b219914860b09d02c3e248739e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9abf9b219914860b09d02c3e248739e" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46689 } } }
I20260810 23:10:03.207773 31787 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:03.207715 31788 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:03.209429 31795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:03.214740 31795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:03.215431 31795 catalog_manager.cc:1259] Loaded cluster ID: 6be10703fe1642d399b19d7aad0d49c3
I20260810 23:10:03.215502 31795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:03.219089 31795 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:03.219882 31795 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Loaded TSK: 0
I20260810 23:10:03.219975 31795 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9abf9b219914860b09d02c3e248739e: Loaded TSK: 1
F20260810 23:10:03.220705 31795 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1f29a499ad google::LogMessage::Fail() at ??:0
@ 0x7f1f29a4c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1f29a494de google::LogMessage::Flush() at ??:0
@ 0x7f1f29a4aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1f2c70d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1f2c705f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1f343ae7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1f3438899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1f34389178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1f343cadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1f343e982b std::function<>::operator()() at ??:0
@ 0x7f1f34388b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1f34389f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1f34386579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1f343ca640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f2bc9b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1f2bc9bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1f2bc9d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1f2bc8c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1f2d2d06db start_thread at ??:0
@ 0x7f1f2854761f clone at ??:0
W20260810 23:10:03.330159 31784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:03.330432 31784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:03.330478 31784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:03.334646 31784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:03.334708 31784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:03.334800 31784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:03.340400 31784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=46651
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:03.341629 31784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:03.342674 31784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:03.353603 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.353603 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.353603 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:03.354157 31784 server_base.cc:1034] running on GCE node
I20260810 23:10:03.354604 31784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:03.355293 31784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:03.356499 31784 hybrid_clock.cc:648] HybridClock initialized: now 1786403403356472 us; error 46 us; skew 500 ppm
Aug 10 23:10:03 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:03.359418 31784 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:10:03.360311 31784 webserver.cc:460] Webserver started at http://127.14.75.1:46651/ using document root <none> and password file <none>
I20260810 23:10:03.360881 31784 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:03.360944 31784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:03.364938 31784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:10:03.367305 31841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.368638 31784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:10:03.368822 31784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "bbb02d18786e4537a4f3bc088873e5ec"
format_stamp: "Formatted at 2026-08-10 23:09:20 on dist-test-slave-srgv"
server_key: "0039ebd66fb5eefdff24647f54d130ed"
server_key_iv: "1f86b721081d0789eab6b965f3b88720"
server_key_version: "encryptionkey@0"
I20260810 23:10:03.369412 31784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:03.390491 31784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:03.394110 31784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:03.394342 31784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:03.395082 31784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:03.396337 31848 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:10:03.402035 31784 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:10:03.402133 31784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 23:10:03.402225 31784 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:10:03.404048 31784 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:10:03.404132 31784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:10:03.404451 31848 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap starting.
I20260810 23:10:03.415791 31784 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:41217
I20260810 23:10:03.415812 31961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:41217 every 8 connection(s)
I20260810 23:10:03.416884 31784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:03.418817 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31784
I20260810 23:10:03.419366 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:33241
--local_ip_for_outbound_sockets=127.14.75.2
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=43777
--webserver_interface=127.14.75.2
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:10:03.427690 31848 log.cc:824] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:03.437120 31848 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:10:03.437619 31848 tablet_bootstrap.cc:492] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Bootstrap complete.
I20260810 23:10:03.439149 31848 ts_tablet_manager.cc:1397] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.011s
I20260810 23:10:03.442384 31848 raft_consensus.cc:348] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:03.442962 31848 raft_consensus.cc:729] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb02d18786e4537a4f3bc088873e5ec, State: Initialized, Role: FOLLOWER
I20260810 23:10:03.443579 31848 consensus_queue.cc:260] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:03.444711 31848 ts_tablet_manager.cc:1428] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260810 23:10:03.463809 31964 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:03.419260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:39155)
0810 23:10:03.420748 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.420823 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:03.422106 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:03.462871 (+ 40765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":174,"threads_started":1}
W20260810 23:10:03.464454 31962 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:46689 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:46689: Client connection negotiation failed: client connection to 127.14.75.62:46689: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:10:03.465328 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.465030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:45599)
0810 23:10:03.465187 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.465264 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:10:03.466045 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.465822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:54017)
0810 23:10:03.465941 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.465991 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:10:03.466270 31962 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:10:03.536734 31965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:03.537002 31965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:03.537046 31965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:03.541143 31965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:03.541204 31965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:03.541265 31965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:03.546583 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:33241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=43777
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:03.547533 31965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:03.548482 31965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:03.558351 31972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.558382 31973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:03.558595 31965 server_base.cc:1034] running on GCE node
W20260810 23:10:03.558733 31975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:03.559130 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:03.559783 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:03.560953 31965 hybrid_clock.cc:648] HybridClock initialized: now 1786403403560923 us; error 40 us; skew 500 ppm
Aug 10 23:10:03 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:03.563791 31965 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:10:03.564677 31965 webserver.cc:460] Webserver started at http://127.14.75.2:43777/ using document root <none> and password file <none>
I20260810 23:10:03.565251 31965 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:03.565347 31965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:03.569111 31965 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:03.571321 31982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.572778 31965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:10:03.572909 31965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "6eafdec34aa443cbbd569d17e9b7c36e"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "a56be787ca273a746503938e1e8baf57"
server_key_iv: "a6639991a00c0fa1f29fd87a88bab160"
server_key_version: "encryptionkey@0"
I20260810 23:10:03.573537 31965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:03.594563 31965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:03.598251 31965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:03.598486 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:03.599184 31965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:03.600081 31965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:03.600165 31965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.600236 31965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:03.600288 31965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.610015 31965 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:33241
I20260810 23:10:03.610052 32100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:33241 every 8 connection(s)
I20260810 23:10:03.611207 31965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:03.614851 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.613313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:60261)
0810 23:10:03.614034 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.614146 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":123,"threads_started":1}
W20260810 23:10:03.615417 32101 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:46689 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:46689: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
I20260810 23:10:03.616022 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.615815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:50839)
0810 23:10:03.615921 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.615969 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:10:03.616647 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.616408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:38261)
0810 23:10:03.616532 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.616585 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 23:10:03.616942 32101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:10:03.617506 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 31965
I20260810 23:10:03.617893 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:45673
--local_ip_for_outbound_sockets=127.14.75.3
--tserver_master_addrs=127.14.75.62:46689
--webserver_port=42929
--webserver_interface=127.14.75.3
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:03.732281 32104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:03.732574 32104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:03.732623 32104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:03.736601 32104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:03.736662 32104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:03.736720 32104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:03.742233 32104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:45673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=42929
--tserver_master_addrs=127.14.75.62:46689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:03.743448 32104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:03.744520 32104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:03.754828 32109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.755131 32110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:03.755285 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:03.755630 32104 server_base.cc:1034] running on GCE node
I20260810 23:10:03.756047 32104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:03.756719 32104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:03.757889 32104 hybrid_clock.cc:648] HybridClock initialized: now 1786403403757865 us; error 36 us; skew 500 ppm
Aug 10 23:10:03 dist-test-slave-srgv krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:03.761081 32104 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:10:03.762108 32104 webserver.cc:460] Webserver started at http://127.14.75.3:42929/ using document root <none> and password file <none>
I20260810 23:10:03.762701 32104 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:03.762809 32104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:03.766837 32104 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:10:03.768956 32120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.770268 32104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:03.770413 32104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "f1e13b5b6bbb426789590cd7458bf3e3"
format_stamp: "Formatted at 2026-08-10 23:09:21 on dist-test-slave-srgv"
server_key: "317fd148172709206ec660e70fb52607"
server_key_iv: "d4b21a7bf5c79b7bc1cbb33419866ebd"
server_key_version: "encryptionkey@0"
I20260810 23:10:03.771019 32104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:03.787907 32104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:03.791360 32104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:03.791581 32104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:03.792200 32104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:03.793063 32104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:03.793147 32104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.793223 32104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:03.793273 32104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:03.802781 32104 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45673
I20260810 23:10:03.802802 32237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45673 every 8 connection(s)
I20260810 23:10:03.803824 32104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:03.805147 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 32104
I20260810 23:10:03.807665 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.806157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:38027)
0810 23:10:03.806863 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.806967 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":122,"threads_started":1}
W20260810 23:10:03.808123 32238 heartbeater.cc:644] Failed to heartbeat to 127.14.75.62:46689 (0 consecutive failures): Network error: Failed to ping master at 127.14.75.62:46689: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
I20260810 23:10:03.808709 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.808504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:50823)
0810 23:10:03.808603 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.808658 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:10:03.809297 32242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:03.809114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:42387)
0810 23:10:03.809209 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:03.809249 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:10:03.809510 32238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:10:04.467386 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.466925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:37681)
0810 23:10:04.467209 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:04.467268 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260810 23:10:04.609553 32244 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:04.609822 32244 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:04.610975 32244 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
I20260810 23:10:04.618040 32248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.617557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:39523)
0810 23:10:04.617838 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:04.617908 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
Aug 10 23:10:04 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.3@KRBTEST.COM
Aug 10 23:10:04 dist-test-slave-srgv krb5kdc[30585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403403, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.2@KRBTEST.COM
I20260810 23:10:04.626533 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.611176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:33241 (local address 127.14.75.1:55677)
0810 23:10:04.611390 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:04.611407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:04.611519 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:04.613636 (+ 2117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:04.613643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:04.614288 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:04.614837 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:04.614851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.616193 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.616199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:04.617211 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:04.617222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.618369 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.618373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:04.618635 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:04.619462 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:04.619474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:04.621462 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:04.623838 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:04.623841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:04.623842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:04.624166 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:04.624595 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:04.624598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:04.624758 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:04.625446 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:04.625453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:04.625737 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 23:10:04.625935 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:10:04.626329 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":227,"spinlock_wait_cycles":56960}
I20260810 23:10:04.626535 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.611709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:57351 (local address 127.14.75.3:45673)
0810 23:10:04.612106 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:04.612116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:04.612146 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:10:04.612882 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:04.612889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:04.613217 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:04.613560 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:04.615047 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.616047 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.617440 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.618119 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.618626 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:04.621397 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:04.621429 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:04.621447 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:04.621483 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:04.623554 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624284 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:04.624291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:04.624297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:04.624432 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624844 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:04.624849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:04.624851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:04.625292 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:04.625455 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:04.626054 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 23:10:04.626333 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":114,"threads_started":1}
I20260810 23:10:04.626561 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.611569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:55677 (local address 127.14.75.2:33241)
0810 23:10:04.611949 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:04.611963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:04.612003 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:10:04.612812 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:04.612820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:04.613231 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:04.614149 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:04.615033 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.616028 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.617485 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.618062 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.618308 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:04.621654 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:04.621659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:04.621669 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:04.621707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:04.623700 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624325 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:04.624330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:04.624334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:04.624455 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624879 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:04.624881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:04.624882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:04.625291 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:04.625459 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:04.626069 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 23:10:04.626363 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":93,"threads_started":1}
I20260810 23:10:04.626922 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.611278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45673 (local address 127.14.75.1:57351)
0810 23:10:04.611729 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:04.611743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:04.611869 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:04.613636 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:04.613643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:04.614289 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:04.614834 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:04.614851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.616237 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.616240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:04.617219 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:04.617229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:04.618377 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:04.618380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:04.618643 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:04.619295 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:04.619314 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:04.621111 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:04.623714 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:04.623721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:04.623736 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:04.624155 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:04.624571 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:04.624577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:04.624581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:04.624743 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:04.625449 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:04.625458 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:04.625737 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 23:10:04.625935 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:10:04.626418 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":373,"spinlock_wait_cycles":19456,"thread_start_us":70,"threads_started":1}
W20260810 23:10:04.629115 31843 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
W20260810 23:10:04.629387 31842 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
I20260810 23:10:04.629503 31842 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb02d18786e4537a4f3bc088873e5ec; no voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:10:04.629788 32244 raft_consensus.cc:2726] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:10:04.810448 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:04.810045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:35043)
0810 23:10:04.810293 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:04.810349 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260810 23:10:05.468626 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:05.468217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:44019)
0810 23:10:05.468480 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:05.468532 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260810 23:10:05.619531 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:05.619113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:36235)
0810 23:10:05.619377 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:05.619439 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260810 23:10:05.811574 32252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:05.811175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:33917)
0810 23:10:05.811414 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:05.811469 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260810 23:10:06.469906 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:06.469499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:44233)
0810 23:10:06.469748 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:06.469807 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260810 23:10:06.620687 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:06.620249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:34051)
0810 23:10:06.620537 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:06.620591 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260810 23:10:06.754212 32255 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:06.754402 32255 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:06.754977 32255 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
W20260810 23:10:06.755893 31842 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
W20260810 23:10:06.756021 31843 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
I20260810 23:10:06.756135 31843 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb02d18786e4537a4f3bc088873e5ec; no voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:10:06.756315 32255 raft_consensus.cc:2726] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:10:06.812808 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:06.812384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:39335)
0810 23:10:06.812649 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:06.812707 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260810 23:10:07.471239 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:07.470754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:49021)
0810 23:10:07.471066 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:07.471138 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260810 23:10:07.621981 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:07.621546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:49915)
0810 23:10:07.621806 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:07.621865 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260810 23:10:07.814296 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:07.813778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:33369)
0810 23:10:07.814132 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:07.814197 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260810 23:10:08.472647 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:08.472116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:39891)
0810 23:10:08.472500 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:08.472555 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":90,"threads_started":1}
I20260810 23:10:08.623134 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:08.622741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:37055)
0810 23:10:08.622987 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:08.623039 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260810 23:10:08.815614 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:08.815167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:37871)
0810 23:10:08.815455 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:08.815518 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260810 23:10:09.473770 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:09.473364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:51119)
0810 23:10:09.473623 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:09.473676 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260810 23:10:09.529503 32265 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:09.529640 32265 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:09.530002 32265 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
W20260810 23:10:09.530828 31843 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
W20260810 23:10:09.530926 31842 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
I20260810 23:10:09.531038 31842 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb02d18786e4537a4f3bc088873e5ec; no voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:10:09.531190 32265 raft_consensus.cc:2726] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:10:09.624356 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:09.623921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:44191)
0810 23:10:09.624199 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:09.624264 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260810 23:10:09.817070 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:09.816632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:48369)
0810 23:10:09.816913 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:09.816974 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260810 23:10:10.475154 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:10.474638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:51121)
0810 23:10:10.474976 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:10.475044 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
I20260810 23:10:10.625873 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:10.625278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:41679)
0810 23:10:10.625710 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:10.625776 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":160,"threads_started":1}
I20260810 23:10:10.818492 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:10.817948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:43943)
0810 23:10:10.818332 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:10.818397 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":94,"threads_started":1}
I20260810 23:10:11.476383 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:11.475949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:43675)
0810 23:10:11.476217 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:11.476286 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260810 23:10:11.627359 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:11.626846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:55017)
0810 23:10:11.627188 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:11.627259 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20260810 23:10:11.735792 32273 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:11.735949 32273 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:11.736694 32273 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
W20260810 23:10:11.737608 31843 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
W20260810 23:10:11.737731 31842 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
I20260810 23:10:11.737777 31843 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb02d18786e4537a4f3bc088873e5ec; no voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:10:11.737918 32273 raft_consensus.cc:2726] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:10:11.819937 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:11.819460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:58185)
0810 23:10:11.819768 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:11.819838 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260810 23:10:12.477689 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:12.477142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:36431)
0810 23:10:12.477495 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:12.477562 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260810 23:10:12.628630 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:12.628190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:46605)
0810 23:10:12.628463 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:12.628521 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260810 23:10:12.821305 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:12.820875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:39979)
0810 23:10:12.821149 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:12.821206 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260810 23:10:13.479049 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:13.478574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:51165)
0810 23:10:13.478882 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:13.478944 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
I20260810 23:10:13.629961 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:13.629537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:48599)
0810 23:10:13.629804 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:13.629862 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260810 23:10:13.822553 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:13.822114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:34863)
0810 23:10:13.822394 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:13.822455 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260810 23:10:14.480156 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:14.479756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:57097)
0810 23:10:14.480006 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:14.480059 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260810 23:10:14.631265 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:14.630815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:50053)
0810 23:10:14.631091 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:14.631165 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260810 23:10:14.823724 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:14.823321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:50915)
0810 23:10:14.823574 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:14.823633 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260810 23:10:15.422053 32285 raft_consensus.cc:482] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:15.422240 32285 raft_consensus.cc:504] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eafdec34aa443cbbd569d17e9b7c36e" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 33241 } } peers { permanent_uuid: "f1e13b5b6bbb426789590cd7458bf3e3" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45673 } } peers { permanent_uuid: "bbb02d18786e4537a4f3bc088873e5ec" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 41217 } }
I20260810 23:10:15.422729 32285 leader_election.cc:288] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241), f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673)
W20260810 23:10:15.423477 31842 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eafdec34aa443cbbd569d17e9b7c36e (127.14.75.2:33241): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
W20260810 23:10:15.423619 31843 leader_election.cc:341] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e13b5b6bbb426789590cd7458bf3e3 (127.14.75.3:45673): Not found: Tablet not found: 13f224a603214116b35e6f03029315b9
I20260810 23:10:15.423740 31843 leader_election.cc:302] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbb02d18786e4537a4f3bc088873e5ec; no voters: 6eafdec34aa443cbbd569d17e9b7c36e, f1e13b5b6bbb426789590cd7458bf3e3
I20260810 23:10:15.423908 32285 raft_consensus.cc:2726] T 13f224a603214116b35e6f03029315b9 P bbb02d18786e4537a4f3bc088873e5ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:10:15.481340 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:15.480924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:43619)
0810 23:10:15.481184 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:15.481245 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260810 23:10:15.632488 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:15.632085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:56207)
0810 23:10:15.632339 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:15.632394 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260810 23:10:15.825062 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:15.824668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:46395)
0810 23:10:15.824911 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:15.824963 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260810 23:10:16.482481 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:16.482076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:46007)
0810 23:10:16.482328 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:16.482385 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260810 23:10:16.633860 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:16.633328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:39167)
0810 23:10:16.633694 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:16.633749 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":96,"threads_started":1}
I20260810 23:10:16.826318 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:16.825893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:37853)
0810 23:10:16.826159 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:16.826215 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260810 23:10:17.483685 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:17.483246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:57163)
0810 23:10:17.483525 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:17.483584 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260810 23:10:17.635022 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:17.634601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:39795)
0810 23:10:17.634872 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:17.634925 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260810 23:10:17.827690 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:17.827249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.3:51197)
0810 23:10:17.827509 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:17.827575 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260810 23:10:18.485287 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:18.484821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.1:41515)
0810 23:10:18.485119 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:18.485189 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260810 23:10:18.636377 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:18.635960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46689 (local address 127.14.75.2:51725)
0810 23:10:18.636225 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:18.636282 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.62:46689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260810 23:10:18.813628 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31784
I20260810 23:10:18.820806 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 31965
I20260810 23:10:18.827603 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 32104
2026-08-10T23:10:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59660 ms)
[----------] 30 tests from SecurityITest (247558 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:18 dist-test-slave-srgv krb5kdc[32301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:18 dist-test-slave-srgv krb5kdc[32301](info): set up 2 sockets
Aug 10 23:10:18 dist-test-slave-srgv krb5kdc[32301](info): commencing operation
krb5kdc: starting...
W20260810 23:10:19.907235 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:19 dist-test-slave-srgv krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:20.036705 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46253
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:46253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:20.148545 32314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:20.148876 32314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:20.148954 32314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:20.153014 32314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:20.153127 32314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:20.153164 32314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:20.153203 32314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:20.153240 32314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:20.158768 32314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:46253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:20.159879 32314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:20.160919 32314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:20.169797 32323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.169864 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.169852 32314 server_base.cc:1034] running on GCE node
W20260810 23:10:20.169857 32321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.170611 32314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:20.171779 32314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:20.172971 32314 hybrid_clock.cc:648] HybridClock initialized: now 1786403420172947 us; error 39 us; skew 500 ppm
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:20.175904 32314 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:20.176934 32314 webserver.cc:460] Webserver started at http://127.14.75.62:43505/ using document root <none> and password file <none>
I20260810 23:10:20.177623 32314 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:20.177697 32314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:20.177871 32314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:20.179862 32314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "c660e3e5b5224138b1306e0aa8ef57c9"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "7d676516c729a46d582de912711f17ea"
server_key_iv: "2c55f12bd68bca552c46d5027021cdf2"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.180375 32314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "c660e3e5b5224138b1306e0aa8ef57c9"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "7d676516c729a46d582de912711f17ea"
server_key_iv: "2c55f12bd68bca552c46d5027021cdf2"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.184005 32314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:10:20.186380 32330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.187564 32314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:20.187670 32314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "c660e3e5b5224138b1306e0aa8ef57c9"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "7d676516c729a46d582de912711f17ea"
server_key_iv: "2c55f12bd68bca552c46d5027021cdf2"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.187760 32314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:20.201030 32314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:20.203955 32314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:20.204140 32314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:20.211484 32314 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:46253
I20260810 23:10:20.211588 32384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:46253 every 8 connection(s)
I20260810 23:10:20.212517 32314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:20.213085 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 32314
I20260810 23:10:20.213233 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:20.213542 14636 external_mini_cluster.cc:1428] Setting key 574d4f3ced038e477207c3385b353dc0
I20260810 23:10:20.216419 32385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:20.221935 32385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: Bootstrap starting.
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:20.224804 32385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:20.225734 32385 log.cc:824] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:20.227962 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.214812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.14.75.62:46253)
0810 23:10:20.215214 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:20.215229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:20.215274 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:10:20.215899 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:20.215914 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:20.216239 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:20.216587 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:20.217459 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.218476 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.219362 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.219834 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.220013 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.222659 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:20.222666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:20.222682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:20.222715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:20.224719 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.225198 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.225204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.225210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.225348 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.225680 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.225684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.225685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.226221 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:20.226378 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:20.226763 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 23:10:20.227052 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260810 23:10:20.228044 32385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: No bootstrap required, opened a new log
I20260810 23:10:20.231844 32385 raft_consensus.cc:348] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } }
I20260810 23:10:20.232055 32385 raft_consensus.cc:374] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:20.232089 32385 raft_consensus.cc:729] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c660e3e5b5224138b1306e0aa8ef57c9, State: Initialized, Role: FOLLOWER
I20260810 23:10:20.232494 32385 consensus_queue.cc:260] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } }
I20260810 23:10:20.232612 32385 raft_consensus.cc:388] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:20.232661 32385 raft_consensus.cc:482] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:20.232729 32385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:20.233644 32385 raft_consensus.cc:504] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } }
I20260810 23:10:20.233927 32385 leader_election.cc:302] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c660e3e5b5224138b1306e0aa8ef57c9; no voters:
I20260810 23:10:20.234196 32385 leader_election.cc:288] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:20.234333 32392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:20.234627 32392 raft_consensus.cc:686] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [term 1 LEADER]: Becoming Leader. State: Replica: c660e3e5b5224138b1306e0aa8ef57c9, State: Running, Role: LEADER
I20260810 23:10:20.235221 32392 consensus_queue.cc:237] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } }
I20260810 23:10:20.235499 32385 sys_catalog.cc:564] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:20.237180 32394 sys_catalog.cc:455] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c660e3e5b5224138b1306e0aa8ef57c9. Latest consensus state: current_term: 1 leader_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } } }
I20260810 23:10:20.237329 32394 sys_catalog.cc:458] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:20.237193 32393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e3e5b5224138b1306e0aa8ef57c9" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 46253 } } }
I20260810 23:10:20.237486 32393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:20.238489 32402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:20.241884 32402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:20.247738 32402 catalog_manager.cc:1347] Generated new cluster ID: e04b22cafebf447290875c0f4895849a
I20260810 23:10:20.247819 32402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:20.279161 32402 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:20.280164 32402 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:20.286679 32402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: Generated new TSK 0
I20260810 23:10:20.287382 32402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:10:20.326704 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:20.443105 32414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:20.443389 32414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:20.443434 32414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:20.447721 32414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:20.447801 32414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:20.447865 32414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:20.453603 32414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:20.454782 32414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:20.455910 32414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:20.465794 32420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.465816 32423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.465853 32421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.466188 32414 server_base.cc:1034] running on GCE node
I20260810 23:10:20.466638 32414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:20.467278 32414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:20.468462 32414 hybrid_clock.cc:648] HybridClock initialized: now 1786403420468444 us; error 33 us; skew 500 ppm
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:20.471326 32414 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:10:20.472261 32414 webserver.cc:460] Webserver started at http://127.14.75.1:45537/ using document root <none> and password file <none>
I20260810 23:10:20.472820 32414 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:20.472909 32414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:20.473155 32414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:20.475278 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "64ed469af06c413a9995f43f429d32cd"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "840edfc9c4a3e78ae2d11076bdb7ce42"
server_key_iv: "3ad0e4a3de7b108cc861e221bccccd77"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.475840 32414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "64ed469af06c413a9995f43f429d32cd"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "840edfc9c4a3e78ae2d11076bdb7ce42"
server_key_iv: "3ad0e4a3de7b108cc861e221bccccd77"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.479678 32414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:10:20.482076 32431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.483302 32414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:20.483443 32414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "64ed469af06c413a9995f43f429d32cd"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "840edfc9c4a3e78ae2d11076bdb7ce42"
server_key_iv: "3ad0e4a3de7b108cc861e221bccccd77"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.483556 32414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:20.497102 32414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:20.500015 32414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:20.500239 32414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:20.500840 32414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:20.501749 32414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:20.501816 32414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.501894 32414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:20.501935 32414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.511372 32414 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:36133
I20260810 23:10:20.511400 32552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:36133 every 8 connection(s)
I20260810 23:10:20.512282 32414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:20.513950 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 32414
I20260810 23:10:20.514104 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:20.514436 14636 external_mini_cluster.cc:1428] Setting key ae24f5e3ee89cda0c8fb3a5c979de468
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:10:20.528651 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.514155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:45557 (local address 127.14.75.62:46253)
0810 23:10:20.514318 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:20.514323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:20.515740 (+ 1417us) server_negotiation.cc:407] Connection header received
0810 23:10:20.517043 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:20.517045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:20.517087 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:20.517156 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:20.518674 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.519483 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.521335 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.521610 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.521693 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.524731 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:20.524735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:20.524737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:20.524761 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:20.526370 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.526939 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.526942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.526943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.527025 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.527339 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.527341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.527342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.527520 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:20.527596 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:20.528272 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 23:10:20.528428 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:10:20.529724 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.514537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46253 (local address 127.14.75.1:45557)
0810 23:10:20.515051 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:20.515110 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:20.516797 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:20.517332 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:20.517343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:20.517940 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:20.518520 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.518534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.519624 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.519630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.521195 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.521209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.521776 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.521783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.522037 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.522736 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:20.522765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:20.524541 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:20.526489 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.526494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.526508 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.526837 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.527110 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.527112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.527113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.527254 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.527650 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:20.527659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:20.527930 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 23:10:20.528183 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 23:10:20.528511 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260810 23:10:20.531095 32553 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46253
I20260810 23:10:20.531373 32553 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:20.531895 32553 heartbeater.cc:511] Master 127.14.75.62:46253 requested a full tablet report, sending...
I20260810 23:10:20.533743 32348 ts_manager.cc:194] Registered new tserver with Master: 64ed469af06c413a9995f43f429d32cd (127.14.75.1:36133)
I20260810 23:10:20.535495 32348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:45557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:10:20.544312 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:20.657159 32559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:20.657454 32559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:20.657498 32559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:20.661688 32559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:20.661753 32559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:20.661811 32559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:20.667250 32559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:20.668394 32559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:20.669476 32559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:20.679191 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.679184 32564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.679206 32565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.679543 32559 server_base.cc:1034] running on GCE node
I20260810 23:10:20.679936 32559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:20.680595 32559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:20.681770 32559 hybrid_clock.cc:648] HybridClock initialized: now 1786403420681752 us; error 30 us; skew 500 ppm
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:20.684652 32559 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:10:20.685658 32559 webserver.cc:460] Webserver started at http://127.14.75.2:40485/ using document root <none> and password file <none>
I20260810 23:10:20.686264 32559 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:20.686359 32559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:20.686591 32559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:20.688602 32559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "3978d804d0a44e22a5718cafa461e4de"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "0ce65e0efa7471cf265e1f1d3fafbc25"
server_key_iv: "6ce897f97136efa034f5d8fbd282ad01"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.689172 32559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "3978d804d0a44e22a5718cafa461e4de"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "0ce65e0efa7471cf265e1f1d3fafbc25"
server_key_iv: "6ce897f97136efa034f5d8fbd282ad01"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.692796 32559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:10:20.695274 32574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.696488 32559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:10:20.696641 32559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "3978d804d0a44e22a5718cafa461e4de"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "0ce65e0efa7471cf265e1f1d3fafbc25"
server_key_iv: "6ce897f97136efa034f5d8fbd282ad01"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.696758 32559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:20.706453 32559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:20.709246 32559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:20.709501 32559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:20.710086 32559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:20.710889 32559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:20.710961 32559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.711026 32559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:20.711086 32559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.720077 32559 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:35481
I20260810 23:10:20.720105 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:35481 every 8 connection(s)
I20260810 23:10:20.720952 32559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:20.721184 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 32559
I20260810 23:10:20.721351 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:20.721803 14636 external_mini_cluster.cc:1428] Setting key 26cc7424d05e5be50c7435371585960f
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:10:20.736838 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.722847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:39919 (local address 127.14.75.62:46253)
0810 23:10:20.722993 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:20.722997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:20.724314 (+ 1317us) server_negotiation.cc:407] Connection header received
0810 23:10:20.725722 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:20.725727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:20.725792 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:20.725894 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:20.727289 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.727878 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.729770 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.729986 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.730093 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.733101 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:20.733105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:20.733108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:20.733139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:20.734642 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.735128 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.735130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.735132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.735248 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.735627 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.735630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.735632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.735866 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:20.735978 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:20.736515 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 23:10:20.736662 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:10:20.737811 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.723308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46253 (local address 127.14.75.2:39919)
0810 23:10:20.724140 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:20.724199 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:20.725464 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:20.726050 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:20.726056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:20.726428 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:20.727107 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.727131 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.728027 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.728033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.729633 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.729649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.730158 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.730165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.730415 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.731088 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:20.731109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:20.732881 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:20.734753 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.734758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.734769 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.735024 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.735374 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.735377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.735378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.735518 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.735978 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:20.735986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:20.736253 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 23:10:20.736454 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 23:10:20.736789 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":125,"threads_started":1}
I20260810 23:10:20.739148 32694 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46253
I20260810 23:10:20.739392 32694 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:20.739893 32694 heartbeater.cc:511] Master 127.14.75.62:46253 requested a full tablet report, sending...
I20260810 23:10:20.741003 32348 ts_manager.cc:194] Registered new tserver with Master: 3978d804d0a44e22a5718cafa461e4de (127.14.75.2:35481)
I20260810 23:10:20.741958 32348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:39919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:10:20.753039 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:20.868918 32699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:20.869166 32699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:20.869210 32699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:20.873231 32699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:20.873292 32699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:20.873350 32699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:20.878788 32699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:20.879729 32699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:20.880668 32699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:20.890372 32707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:20.890431 32704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.890630 32699 server_base.cc:1034] running on GCE node
W20260810 23:10:20.890431 32705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:20.891144 32699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:20.891865 32699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:20.893049 32699 hybrid_clock.cc:648] HybridClock initialized: now 1786403420893035 us; error 37 us; skew 500 ppm
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:20.895849 32699 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:10:20.896760 32699 webserver.cc:460] Webserver started at http://127.14.75.3:44725/ using document root <none> and password file <none>
I20260810 23:10:20.897338 32699 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:20.897459 32699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:20.897703 32699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:20.900112 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "ff92bcfcadb14e599058e236cfe292c5"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "5e4b6396f4192f3fcaca79e155f3960e"
server_key_iv: "1724a259dfd48c7bdc111d0bbb932ecb"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.900744 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "ff92bcfcadb14e599058e236cfe292c5"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "5e4b6396f4192f3fcaca79e155f3960e"
server_key_iv: "1724a259dfd48c7bdc111d0bbb932ecb"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.904603 32699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 23:10:20.907176 32714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.908391 32699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:20.908529 32699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "ff92bcfcadb14e599058e236cfe292c5"
format_stamp: "Formatted at 2026-08-10 23:10:20 on dist-test-slave-srgv"
server_key: "5e4b6396f4192f3fcaca79e155f3960e"
server_key_iv: "1724a259dfd48c7bdc111d0bbb932ecb"
server_key_version: "encryptionkey@0"
I20260810 23:10:20.908648 32699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:20.931157 32699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:20.934679 32699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:20.934924 32699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:20.935588 32699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:20.936522 32699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:20.936597 32699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.936664 32699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:20.936724 32699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:20.946318 32699 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:36357
I20260810 23:10:20.946350 364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:36357 every 8 connection(s)
I20260810 23:10:20.947239 32699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:20.950039 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 32699
I20260810 23:10:20.950227 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:20.950560 14636 external_mini_cluster.cc:1428] Setting key 746149bcde330515e0e053cb7fd9bc24
Aug 10 23:10:20 dist-test-slave-srgv krb5kdc[32301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403420, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:20.961674 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.949049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:48869 (local address 127.14.75.62:46253)
0810 23:10:20.949189 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:20.949203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:20.950159 (+ 956us) server_negotiation.cc:407] Connection header received
0810 23:10:20.951841 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:20.951845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:20.951909 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:20.951993 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:20.953352 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.953913 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.955251 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.955461 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.955539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.958219 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:20.958222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:20.958224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:20.958247 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:20.959722 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.960222 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.960225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.960226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.960302 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.960621 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.960623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.960624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.960817 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:20.960895 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:20.961397 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 23:10:20.961553 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:10:20.962555 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.949433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:46253 (local address 127.14.75.3:48869)
0810 23:10:20.949979 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:20.950044 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:20.951556 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:20.952204 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:20.952211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:20.952604 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:20.953213 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.953229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.954096 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.954100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.955131 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:20.955142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.955620 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.955624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:20.955772 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.956330 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:20.956365 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:20.958022 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:20.959849 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.959853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.959865 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.960133 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.960383 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:20.960385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:20.960386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:20.960541 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:20.960912 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:20.960917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:20.961188 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 23:10:20.961351 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 23:10:20.961690 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":120,"threads_started":1}
I20260810 23:10:20.963858 365 heartbeater.cc:347] Connected to a master server at 127.14.75.62:46253
I20260810 23:10:20.964138 365 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:20.964746 365 heartbeater.cc:511] Master 127.14.75.62:46253 requested a full tablet report, sending...
I20260810 23:10:20.965855 32348 ts_manager.cc:194] Registered new tserver with Master: ff92bcfcadb14e599058e236cfe292c5 (127.14.75.3:36357)
I20260810 23:10:20.966424 32348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:48869
I20260810 23:10:20.976835 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:20.986725 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:20.979130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.14.75.62:46253)
0810 23:10:20.979265 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:20.979272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:20.979451 (+ 179us) server_negotiation.cc:407] Connection header received
0810 23:10:20.979611 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:20.979614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:20.979672 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:20.979765 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:20.980488 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.981016 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.981984 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:20.982218 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:20.982364 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:20.983772 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:20.983778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:20.983786 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:20.983811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:20.984984 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.985446 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.985451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.985457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.985534 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:20.985819 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:20.985823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:20.985825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:20.986051 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:20.986194 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:20.986428 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 23:10:20.986547 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:10:20.989779 32348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:20.993026 32348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:21.008327 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.002430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.14.75.1:36133)
0810 23:10:21.003396 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.003415 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.003440 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:10:21.003518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.003538 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.003733 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.003909 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:21.004187 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.005433 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.006972 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.007551 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.007682 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.007846 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.008068 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.008179 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":87,"threads_started":1}
I20260810 23:10:21.008320 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.002054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:36133 (local address 127.0.0.1:56284)
0810 23:10:21.002704 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.002755 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:21.002897 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:21.003900 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:21.003905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:21.003921 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:21.004058 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.004069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.005600 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.005603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.006836 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.006846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.007683 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.007686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.007844 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.007904 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:10:21.008027 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:10:21.008153 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":30,"spinlock_wait_cycles":5632,"thread_start_us":145,"threads_started":1}
I20260810 23:10:21.009191 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.001807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36357 (local address 127.0.0.1:36540)
0810 23:10:21.002680 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.002732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:21.002896 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:21.003594 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:21.003601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:21.003626 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:21.003796 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.003808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.005775 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.005778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.007267 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.007279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.008680 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.008686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.008789 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.008818 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:10:21.008901 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:10:21.008986 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":45,"spinlock_wait_cycles":3456,"thread_start_us":162,"threads_started":1}
I20260810 23:10:21.009946 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.003088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35481 (local address 127.0.0.1:51042)
0810 23:10:21.003993 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.004009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:21.004117 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:21.004952 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:21.004955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:21.004972 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:21.005092 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.005099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.006483 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.006487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.007815 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.007826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.009460 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.009464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.009604 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.009637 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:10:21.009716 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:10:21.009793 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":26,"thread_start_us":88,"threads_started":1}
I20260810 23:10:21.010197 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.001996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.14.75.3:36357)
0810 23:10:21.002390 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.002408 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.003052 (+ 644us) server_negotiation.cc:407] Connection header received
0810 23:10:21.003140 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.003148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.003371 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.003607 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:21.003951 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.005388 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.007424 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.008353 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.008463 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.008603 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.009727 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.009831 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":163,"threads_started":1}
I20260810 23:10:21.010450 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.003523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.14.75.2:35481)
0810 23:10:21.004351 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.004360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.004377 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:10:21.004434 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.004438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.004757 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.004957 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:21.005207 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.006328 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.007964 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.009083 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.009197 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.009367 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.010156 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.010282 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":110,"threads_started":1}
I20260810 23:10:21.011663 32480 tablet_service.cc:1467] Processing CreateTablet for tablet 036d1d3a0a1a4786b1eaf43e8eea028c (DEFAULT_TABLE table=test-table [id=7e39a39b708849e3a4c59c2675eac5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:21.012738 32480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036d1d3a0a1a4786b1eaf43e8eea028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:21.012856 32764 tablet_service.cc:1467] Processing CreateTablet for tablet 036d1d3a0a1a4786b1eaf43e8eea028c (DEFAULT_TABLE table=test-table [id=7e39a39b708849e3a4c59c2675eac5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:21.013971 32764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036d1d3a0a1a4786b1eaf43e8eea028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:21.019971 384 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Bootstrap starting.
I20260810 23:10:21.020044 32625 tablet_service.cc:1467] Processing CreateTablet for tablet 036d1d3a0a1a4786b1eaf43e8eea028c (DEFAULT_TABLE table=test-table [id=7e39a39b708849e3a4c59c2675eac5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:21.020691 383 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Bootstrap starting.
I20260810 23:10:21.021203 32625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 036d1d3a0a1a4786b1eaf43e8eea028c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:21.022698 384 tablet_bootstrap.cc:654] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:21.022868 383 tablet_bootstrap.cc:654] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:21.023917 384 log.cc:824] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:21.024466 383 log.cc:824] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:21.026371 383 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: No bootstrap required, opened a new log
I20260810 23:10:21.026458 384 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: No bootstrap required, opened a new log
I20260810 23:10:21.026595 383 ts_tablet_manager.cc:1397] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:10:21.026659 384 ts_tablet_manager.cc:1397] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:10:21.028877 387 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Bootstrap starting.
I20260810 23:10:21.029783 383 raft_consensus.cc:348] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.030079 383 raft_consensus.cc:374] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:21.030181 383 raft_consensus.cc:729] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff92bcfcadb14e599058e236cfe292c5, State: Initialized, Role: FOLLOWER
I20260810 23:10:21.030645 383 consensus_queue.cc:260] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.030714 384 raft_consensus.cc:348] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.031005 384 raft_consensus.cc:374] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:21.031095 384 raft_consensus.cc:729] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ed469af06c413a9995f43f429d32cd, State: Initialized, Role: FOLLOWER
I20260810 23:10:21.031574 384 consensus_queue.cc:260] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.031666 387 tablet_bootstrap.cc:654] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:21.031749 383 ts_tablet_manager.cc:1428] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:10:21.032142 365 heartbeater.cc:503] Master 127.14.75.62:46253 was elected leader, sending a full tablet report...
I20260810 23:10:21.032441 32553 heartbeater.cc:503] Master 127.14.75.62:46253 was elected leader, sending a full tablet report...
I20260810 23:10:21.032517 387 log.cc:824] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:21.033360 384 ts_tablet_manager.cc:1428] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:10:21.036113 387 tablet_bootstrap.cc:492] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: No bootstrap required, opened a new log
I20260810 23:10:21.036330 387 ts_tablet_manager.cc:1397] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:10:21.039871 387 raft_consensus.cc:348] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.040127 387 raft_consensus.cc:374] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:21.040207 387 raft_consensus.cc:729] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3978d804d0a44e22a5718cafa461e4de, State: Initialized, Role: FOLLOWER
I20260810 23:10:21.040675 387 consensus_queue.cc:260] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.041337 32694 heartbeater.cc:503] Master 127.14.75.62:46253 was elected leader, sending a full tablet report...
I20260810 23:10:21.041676 387 ts_tablet_manager.cc:1428] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 23:10:21.198237 366 tablet.cc:2369] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:10:21.222259 32695 tablet.cc:2369] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:10:21.263638 32555 tablet.cc:2369] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:10:21.477793 389 raft_consensus.cc:482] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:21.478024 389 raft_consensus.cc:504] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.479374 389 leader_election.cc:288] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff92bcfcadb14e599058e236cfe292c5 (127.14.75.3:36357), 3978d804d0a44e22a5718cafa461e4de (127.14.75.2:35481)
I20260810 23:10:21.485117 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.479766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:59169 (local address 127.14.75.2:35481)
0810 23:10:21.479895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.479900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.480302 (+ 402us) server_negotiation.cc:407] Connection header received
0810 23:10:21.480376 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.480379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.480437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.480531 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:21.481058 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.481901 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.483775 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.484414 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.484541 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.484602 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.484914 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.485012 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:10:21.485163 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.479649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35481 (local address 127.14.75.1:59169)
0810 23:10:21.480198 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.480211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:21.480287 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:21.480557 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:21.480560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:21.480601 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:21.480923 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.480931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.482037 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.482040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.483629 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.483638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.484561 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.484565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.484737 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.484775 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:10:21.484868 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:10:21.484983 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":331,"spinlock_wait_cycles":7424,"thread_start_us":102,"threads_started":1}
I20260810 23:10:21.485189 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.479759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:52743 (local address 127.14.75.3:36357)
0810 23:10:21.479895 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.479900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.480246 (+ 346us) server_negotiation.cc:407] Connection header received
0810 23:10:21.480388 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.480391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.480442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.480530 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:21.481058 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.481865 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.483775 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.484413 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.484532 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.484602 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.484986 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.485081 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:10:21.485615 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.479642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36357 (local address 127.14.75.1:52743)
0810 23:10:21.480118 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.480134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:21.480278 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:21.480557 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:21.480560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:21.480600 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:21.480923 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.480931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.482037 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.482040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.483625 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:21.483638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.484561 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.484565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:21.484741 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.484790 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:10:21.485333 (+ 543us) client_negotiation.cc:240] Negotiation successful
0810 23:10:21.485456 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":244,"spinlock_wait_cycles":30720,"thread_start_us":143,"threads_started":1}
I20260810 23:10:21.485967 32646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036d1d3a0a1a4786b1eaf43e8eea028c" candidate_uuid: "64ed469af06c413a9995f43f429d32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3978d804d0a44e22a5718cafa461e4de" is_pre_election: true
I20260810 23:10:21.486224 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036d1d3a0a1a4786b1eaf43e8eea028c" candidate_uuid: "64ed469af06c413a9995f43f429d32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff92bcfcadb14e599058e236cfe292c5" is_pre_election: true
I20260810 23:10:21.486370 32646 raft_consensus.cc:2454] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ed469af06c413a9995f43f429d32cd in term 0.
I20260810 23:10:21.486513 317 raft_consensus.cc:2454] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ed469af06c413a9995f43f429d32cd in term 0.
I20260810 23:10:21.486977 32435 leader_election.cc:302] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3978d804d0a44e22a5718cafa461e4de, 64ed469af06c413a9995f43f429d32cd; no voters:
I20260810 23:10:21.487301 389 raft_consensus.cc:2781] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:21.487398 389 raft_consensus.cc:482] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:21.487442 389 raft_consensus.cc:3037] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:21.488584 389 raft_consensus.cc:504] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.488968 389 leader_election.cc:288] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [CANDIDATE]: Term 1 election: Requested vote from peers ff92bcfcadb14e599058e236cfe292c5 (127.14.75.3:36357), 3978d804d0a44e22a5718cafa461e4de (127.14.75.2:35481)
I20260810 23:10:21.489388 317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036d1d3a0a1a4786b1eaf43e8eea028c" candidate_uuid: "64ed469af06c413a9995f43f429d32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff92bcfcadb14e599058e236cfe292c5"
I20260810 23:10:21.489449 32646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "036d1d3a0a1a4786b1eaf43e8eea028c" candidate_uuid: "64ed469af06c413a9995f43f429d32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3978d804d0a44e22a5718cafa461e4de"
I20260810 23:10:21.489601 317 raft_consensus.cc:3037] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:21.489614 32646 raft_consensus.cc:3037] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:21.490850 32646 raft_consensus.cc:2454] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ed469af06c413a9995f43f429d32cd in term 1.
I20260810 23:10:21.490865 317 raft_consensus.cc:2454] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ed469af06c413a9995f43f429d32cd in term 1.
I20260810 23:10:21.491197 32435 leader_election.cc:302] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3978d804d0a44e22a5718cafa461e4de, 64ed469af06c413a9995f43f429d32cd; no voters:
I20260810 23:10:21.491418 389 raft_consensus.cc:2781] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:21.491709 389 raft_consensus.cc:686] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [term 1 LEADER]: Becoming Leader. State: Replica: 64ed469af06c413a9995f43f429d32cd, State: Running, Role: LEADER
I20260810 23:10:21.492075 389 consensus_queue.cc:237] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } }
I20260810 23:10:21.495514 32347 catalog_manager.cc:5495] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ed469af06c413a9995f43f429d32cd (127.14.75.1). New cstate: current_term: 1 leader_uuid: "64ed469af06c413a9995f43f429d32cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64ed469af06c413a9995f43f429d32cd" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:21.569952 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.566181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.14.75.1:36133)
0810 23:10:21.566416 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.566419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.566435 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:10:21.566569 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.566571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.566635 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.566753 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:21.567002 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.567554 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.568512 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.568740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.568817 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.569109 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:21.569605 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:21.569703 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.569767 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.569816 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260810 23:10:21.575942 32646 raft_consensus.cc:1261] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 1 FOLLOWER]: Refusing update from remote peer 64ed469af06c413a9995f43f429d32cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:21.575943 317 raft_consensus.cc:1261] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 1 FOLLOWER]: Refusing update from remote peer 64ed469af06c413a9995f43f429d32cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:21.576660 389 consensus_queue.cc:1035] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3978d804d0a44e22a5718cafa461e4de" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:21.576879 394 consensus_queue.cc:1035] T 036d1d3a0a1a4786b1eaf43e8eea028c P 64ed469af06c413a9995f43f429d32cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff92bcfcadb14e599058e236cfe292c5" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:21.592777 32347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:21.592975 32347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:21.595568 32347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c660e3e5b5224138b1306e0aa8ef57c9: Sending DeleteTablet for 3 replicas of tablet 036d1d3a0a1a4786b1eaf43e8eea028c
I20260810 23:10:21.596444 32764 tablet_service.cc:1514] Processing DeleteTablet for tablet 036d1d3a0a1a4786b1eaf43e8eea028c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:21 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:36540
I20260810 23:10:21.596479 32625 tablet_service.cc:1514] Processing DeleteTablet for tablet 036d1d3a0a1a4786b1eaf43e8eea028c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:21 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:51042
I20260810 23:10:21.597098 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 32414
I20260810 23:10:21.597177 411 tablet_replica.cc:330] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: stopping tablet replica
I20260810 23:10:21.598569 411 raft_consensus.cc:2229] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:21.598922 411 raft_consensus.cc:2258] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:21.599785 412 tablet_replica.cc:330] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: stopping tablet replica
I20260810 23:10:21.600199 412 raft_consensus.cc:2229] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:21.600454 411 ts_tablet_manager.cc:1905] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:21.600534 412 raft_consensus.cc:2258] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:21.602206 412 ts_tablet_manager.cc:1905] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:21.603278 411 ts_tablet_manager.cc:1918] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:21.603401 411 log.cc:1196] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/036d1d3a0a1a4786b1eaf43e8eea028c
I20260810 23:10:21.603703 411 ts_tablet_manager.cc:1939] T 036d1d3a0a1a4786b1eaf43e8eea028c P ff92bcfcadb14e599058e236cfe292c5: Deleting consensus metadata
I20260810 23:10:21.604950 32332 catalog_manager.cc:4841] TS ff92bcfcadb14e599058e236cfe292c5 (127.14.75.3:36357): tablet 036d1d3a0a1a4786b1eaf43e8eea028c (table test-table [id=7e39a39b708849e3a4c59c2675eac5ff]) successfully deleted
I20260810 23:10:21.605795 412 ts_tablet_manager.cc:1918] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:21.605911 412 log.cc:1196] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/036d1d3a0a1a4786b1eaf43e8eea028c
I20260810 23:10:21.606261 412 ts_tablet_manager.cc:1939] T 036d1d3a0a1a4786b1eaf43e8eea028c P 3978d804d0a44e22a5718cafa461e4de: Deleting consensus metadata
W20260810 23:10:21.607149 32335 connection.cc:535] client connection to 127.14.75.1:36133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:10:21.607316 32335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:10:21.607540 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 32559
I20260810 23:10:21.610745 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.608065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:36133 (local address 127.0.0.1:56292)
0810 23:10:21.610499 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.610605 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2324,"thread_start_us":244,"threads_started":1}
W20260810 23:10:21.610915 32335 catalog_manager.cc:4584] TS 64ed469af06c413a9995f43f429d32cd (127.14.75.1:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 036d1d3a0a1a4786b1eaf43e8eea028c: Network error: Client connection negotiation failed: client connection to 127.14.75.1:36133: connect: Connection refused (error 111)
I20260810 23:10:21.615231 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 32699
I20260810 23:10:21.618010 414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.615536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:35481 (local address 127.0.0.1:51056)
0810 23:10:21.617818 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:21.617892 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.2:35481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2158}
I20260810 23:10:21.622551 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 32314
2026-08-10T23:10:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:21Z Disabled control of system clock
I20260810 23:10:21.655166 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43779
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:43779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:21.767319 418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:21.767616 418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:21.767653 418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:21.772092 418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:21.772162 418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:21.772177 418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:21.772195 418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:21.772210 418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:21.777179 418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:43779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:43779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:21.778391 418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:21.779953 418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:21.788851 426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:21.788798 424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:21.788827 423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:21.788803 418 server_base.cc:1034] running on GCE node
I20260810 23:10:21.789796 418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:21.790920 418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:21.792101 418 hybrid_clock.cc:648] HybridClock initialized: now 1786403421792073 us; error 44 us; skew 500 ppm
I20260810 23:10:21.794144 418 webserver.cc:460] Webserver started at http://127.14.75.62:38349/ using document root <none> and password file <none>
I20260810 23:10:21.794739 418 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:21.794808 418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:21.794991 418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:21.797188 418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "5cb50e7f264440c2ae96377490ecb7ed"
format_stamp: "Formatted at 2026-08-10 23:10:21 on dist-test-slave-srgv"
server_key: "a0065cb89652e0fdeb8afba35d78a4ff"
server_key_iv: "f3ca26b8f56184e140f415825a86ac18"
server_key_version: "encryptionkey@0"
I20260810 23:10:21.797832 418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "5cb50e7f264440c2ae96377490ecb7ed"
format_stamp: "Formatted at 2026-08-10 23:10:21 on dist-test-slave-srgv"
server_key: "a0065cb89652e0fdeb8afba35d78a4ff"
server_key_iv: "f3ca26b8f56184e140f415825a86ac18"
server_key_version: "encryptionkey@0"
I20260810 23:10:21.802143 418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:10:21.804663 432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:21.806018 418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 23:10:21.806138 418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "5cb50e7f264440c2ae96377490ecb7ed"
format_stamp: "Formatted at 2026-08-10 23:10:21 on dist-test-slave-srgv"
server_key: "a0065cb89652e0fdeb8afba35d78a4ff"
server_key_iv: "f3ca26b8f56184e140f415825a86ac18"
server_key_version: "encryptionkey@0"
I20260810 23:10:21.806241 418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:21.826937 418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:21.827639 418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:21.827806 418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:21.835238 418 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:43779
I20260810 23:10:21.835266 486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:43779 every 8 connection(s)
I20260810 23:10:21.836272 418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:21.839516 487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:21.841586 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 418
I20260810 23:10:21.841738 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:21.842033 14636 external_mini_cluster.cc:1428] Setting key 8a2c7692bc78cad7c1a0d18977528ed5
I20260810 23:10:21.845883 487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: Bootstrap starting.
I20260810 23:10:21.848047 487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:21.848820 487 log.cc:824] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:21.851307 487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: No bootstrap required, opened a new log
I20260810 23:10:21.851984 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:21.843557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.14.75.62:43779)
0810 23:10:21.844226 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:21.844242 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:21.844279 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:10:21.844995 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:21.845014 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:21.845516 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:21.845861 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:21.846335 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.847291 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.848175 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:21.848663 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:21.848853 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:21.849832 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:21.849841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:21.849861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:21.850343 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:21.850413 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:21.850443 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:21.850563 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:21.850705 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:10:21.850988 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":143,"threads_started":1}
I20260810 23:10:21.855314 487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } }
I20260810 23:10:21.855533 487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:21.855566 487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb50e7f264440c2ae96377490ecb7ed, State: Initialized, Role: FOLLOWER
I20260810 23:10:21.855921 487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } }
I20260810 23:10:21.856029 487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:21.856073 487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:21.856149 487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:21.857020 487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } }
I20260810 23:10:21.857452 487 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb50e7f264440c2ae96377490ecb7ed; no voters:
I20260810 23:10:21.857788 487 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:21.857949 492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:21.858193 492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [term 1 LEADER]: Becoming Leader. State: Replica: 5cb50e7f264440c2ae96377490ecb7ed, State: Running, Role: LEADER
I20260810 23:10:21.858529 492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } }
I20260810 23:10:21.859093 487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:21.860244 493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb50e7f264440c2ae96377490ecb7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } } }
I20260810 23:10:21.860379 493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:21.860695 494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb50e7f264440c2ae96377490ecb7ed. Latest consensus state: current_term: 1 leader_uuid: "5cb50e7f264440c2ae96377490ecb7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb50e7f264440c2ae96377490ecb7ed" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 43779 } } }
I20260810 23:10:21.860764 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:21.860795 494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:21.863969 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:21.869907 501 catalog_manager.cc:1347] Generated new cluster ID: ecbf11f5148644309b8253991d9e695c
I20260810 23:10:21.869988 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:21.889696 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:21.890972 501 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:21.897641 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: Generated new TSK 0
I20260810 23:10:21.898458 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:21.903193 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--builtin_ntp_servers=127.14.75.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:22.023547 512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:22.023988 512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:22.024073 512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:22.028350 512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:22.028441 512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:22.028538 512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:22.033345 512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:22.034538 512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:22.035966 512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:22.045801 519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:22.045854 518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:22.045795 521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:22.046192 512 server_base.cc:1034] running on GCE node
I20260810 23:10:22.046607 512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:22.047304 512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:22.048467 512 hybrid_clock.cc:648] HybridClock initialized: now 1786403422048440 us; error 44 us; skew 500 ppm
I20260810 23:10:22.050460 512 webserver.cc:460] Webserver started at http://127.14.75.1:40187/ using document root <none> and password file <none>
I20260810 23:10:22.051054 512 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:22.051118 512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:22.051388 512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:22.053556 512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "752ac632422846cdb1318fbb1037c3f3"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "4b8cf2e5c39c85e3b296ac42e38b6bed"
server_key_iv: "76bf540925e25bfde77c7d12bba4283f"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.054229 512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "752ac632422846cdb1318fbb1037c3f3"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "4b8cf2e5c39c85e3b296ac42e38b6bed"
server_key_iv: "76bf540925e25bfde77c7d12bba4283f"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.057973 512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:10:22.060627 527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.061964 512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:22.062094 512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "752ac632422846cdb1318fbb1037c3f3"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "4b8cf2e5c39c85e3b296ac42e38b6bed"
server_key_iv: "76bf540925e25bfde77c7d12bba4283f"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.062211 512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:22.081260 512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:22.082012 512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:22.082224 512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:22.082856 512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:22.083827 512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:22.083880 512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.083953 512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:22.083986 512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.093168 512 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:35241
I20260810 23:10:22.093194 645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:35241 every 8 connection(s)
I20260810 23:10:22.094179 512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:22.100507 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 512
I20260810 23:10:22.100638 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:22.100932 14636 external_mini_cluster.cc:1428] Setting key 61a6d8cfe9b6afc998bc8668c9a141c7
I20260810 23:10:22.103246 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--builtin_ntp_servers=127.14.75.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:22.104017 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.095974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:46061 (local address 127.14.75.62:43779)
0810 23:10:22.096151 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.096156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.097199 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 23:10:22.098530 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.098533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.098581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.098707 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:22.099592 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.100120 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.101808 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.102103 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.102210 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.103496 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:22.103502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:22.103507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:22.103524 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:22.103560 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:22.103573 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:22.103622 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.103858 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.103911 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:10:22.105195 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.096392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43779 (local address 127.14.75.1:46061)
0810 23:10:22.097031 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.097082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.098345 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.098901 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.098908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.098988 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:22.099467 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.099479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.100279 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.100285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.101672 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.101683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.102284 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.102291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.102496 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.103171 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:22.103173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:22.103225 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:22.103311 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:22.103692 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:22.103705 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:22.103714 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.103838 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.104167 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260810 23:10:22.106565 647 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43779
I20260810 23:10:22.106820 647 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:22.107440 647 heartbeater.cc:511] Master 127.14.75.62:43779 requested a full tablet report, sending...
I20260810 23:10:22.109076 451 ts_manager.cc:194] Registered new tserver with Master: 752ac632422846cdb1318fbb1037c3f3 (127.14.75.1:35241)
I20260810 23:10:22.110366 451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.1:46061
W20260810 23:10:22.233670 650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:22.233943 650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:22.233983 650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:22.238068 650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:22.238128 650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:22.238229 650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:22.243016 650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:22.244118 650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:22.245581 650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:22.255311 655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:22.255311 658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:22.255368 656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:22.255698 650 server_base.cc:1034] running on GCE node
I20260810 23:10:22.256109 650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:22.256801 650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:22.257999 650 hybrid_clock.cc:648] HybridClock initialized: now 1786403422257979 us; error 32 us; skew 500 ppm
I20260810 23:10:22.260145 650 webserver.cc:460] Webserver started at http://127.14.75.2:39631/ using document root <none> and password file <none>
I20260810 23:10:22.260762 650 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:22.260828 650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:22.261090 650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:22.263204 650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "fa26e75c128a4e099186de722afb889c"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "5a1708037e5e531fbd8dd6beee4d8950"
server_key_iv: "e7426b868bf7d196cf43bf481dda40e0"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.263792 650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "fa26e75c128a4e099186de722afb889c"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "5a1708037e5e531fbd8dd6beee4d8950"
server_key_iv: "e7426b868bf7d196cf43bf481dda40e0"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.267681 650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:10:22.270120 664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.271385 650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:22.271533 650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "fa26e75c128a4e099186de722afb889c"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "5a1708037e5e531fbd8dd6beee4d8950"
server_key_iv: "e7426b868bf7d196cf43bf481dda40e0"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.271665 650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:22.301721 650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:22.302465 650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:22.302688 650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:22.303313 650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:22.304332 650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:22.304409 650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.304479 650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:22.304529 650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.313998 650 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:45371
I20260810 23:10:22.314019 781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:45371 every 8 connection(s)
I20260810 23:10:22.314919 650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:22.321197 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 650
I20260810 23:10:22.321314 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:22.321640 14636 external_mini_cluster.cc:1428] Setting key 703d22295474793597a7fc94c467a37a
I20260810 23:10:22.323809 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--builtin_ntp_servers=127.14.75.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:22.325325 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.316705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:45053 (local address 127.14.75.62:43779)
0810 23:10:22.316845 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.316849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.318011 (+ 1162us) server_negotiation.cc:407] Connection header received
0810 23:10:22.319590 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.319594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.319658 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.319770 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:22.320767 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.321620 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.323216 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.323458 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.323544 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.324870 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:22.324873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:22.324875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:22.324886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:22.324905 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:22.324913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:22.324973 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.325164 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.325214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:10:22.326380 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.317097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43779 (local address 127.14.75.2:45053)
0810 23:10:22.317836 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.317880 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.319347 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.319962 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.319969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.320057 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:22.320570 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.320583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.321855 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.321859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.323076 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.323087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.323631 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.323638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.323854 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.324551 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:22.324556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:22.324617 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:22.324705 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:22.325001 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:22.325013 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:22.325023 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.325157 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.325525 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 23:10:22.327581 782 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43779
I20260810 23:10:22.327826 782 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:22.328361 782 heartbeater.cc:511] Master 127.14.75.62:43779 requested a full tablet report, sending...
I20260810 23:10:22.329473 451 ts_manager.cc:194] Registered new tserver with Master: fa26e75c128a4e099186de722afb889c (127.14.75.2:45371)
I20260810 23:10:22.330036 451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.2:45053
W20260810 23:10:22.442270 785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:22.442574 785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:22.442654 785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:22.446817 785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:22.446884 785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:22.446982 785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:22.452200 785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:43779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:22.453300 785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:22.454717 785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:22.464480 790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:22.464650 785 server_base.cc:1034] running on GCE node
W20260810 23:10:22.464522 793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:22.464679 791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:22.465186 785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:22.465955 785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:22.467123 785 hybrid_clock.cc:648] HybridClock initialized: now 1786403422467092 us; error 46 us; skew 500 ppm
I20260810 23:10:22.468972 785 webserver.cc:460] Webserver started at http://127.14.75.3:42383/ using document root <none> and password file <none>
I20260810 23:10:22.469624 785 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:22.469692 785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:22.469950 785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:22.471993 785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "bc96d085772b43f7a0136ab23c4f8d5b"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "cf3633db4000a13e3569af3b00b24921"
server_key_iv: "194fbcb7e7c958bba0ead4c18448c18c"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.472631 785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "bc96d085772b43f7a0136ab23c4f8d5b"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "cf3633db4000a13e3569af3b00b24921"
server_key_iv: "194fbcb7e7c958bba0ead4c18448c18c"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.476837 785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:10:22.479386 800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.480623 785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:10:22.480751 785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "bc96d085772b43f7a0136ab23c4f8d5b"
format_stamp: "Formatted at 2026-08-10 23:10:22 on dist-test-slave-srgv"
server_key: "cf3633db4000a13e3569af3b00b24921"
server_key_iv: "194fbcb7e7c958bba0ead4c18448c18c"
server_key_version: "encryptionkey@0"
I20260810 23:10:22.480868 785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:22.493772 785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:22.494488 785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:22.494725 785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:22.495352 785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:22.496320 785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:22.496393 785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.496465 785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:22.496508 785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:22.505362 785 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:45343
I20260810 23:10:22.505476 919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:45343 every 8 connection(s)
I20260810 23:10:22.506316 785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:22.511804 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 785
I20260810 23:10:22.511920 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:22.512198 14636 external_mini_cluster.cc:1428] Setting key e51c19f16a2a8b141f4385112a98630b
I20260810 23:10:22.516430 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.508157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:46539 (local address 127.14.75.62:43779)
0810 23:10:22.508302 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.508306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.509524 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 23:10:22.511092 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.511094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.511151 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.511229 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:22.512408 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.513092 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.514685 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.514903 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.514986 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.516032 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:22.516035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:22.516037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:22.516048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:22.516066 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:22.516073 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:22.516137 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.516269 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.516319 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:10:22.517304 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.508563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:43779 (local address 127.14.75.3:46539)
0810 23:10:22.509292 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.509366 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.510890 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.511362 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.511368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.511451 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:22.512191 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.512214 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.513286 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.513289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.514571 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.514581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.515037 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.515040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.515178 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.515783 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:22.515784 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:22.515826 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:22.515893 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:22.516147 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:22.516155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:22.516159 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.516254 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.516482 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":120,"threads_started":1}
I20260810 23:10:22.518443 920 heartbeater.cc:347] Connected to a master server at 127.14.75.62:43779
I20260810 23:10:22.518658 920 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:22.519146 920 heartbeater.cc:511] Master 127.14.75.62:43779 requested a full tablet report, sending...
I20260810 23:10:22.520099 451 ts_manager.cc:194] Registered new tserver with Master: bc96d085772b43f7a0136ab23c4f8d5b (127.14.75.3:45343)
I20260810 23:10:22.520673 451 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.3:46539
I20260810 23:10:22.526152 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:22.531586 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.528067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.14.75.62:43779)
0810 23:10:22.528228 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.528233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.528450 (+ 217us) server_negotiation.cc:407] Connection header received
0810 23:10:22.528532 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.528534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.528573 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.528641 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:22.528885 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.529381 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.530198 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.530411 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.530488 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.531262 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:22.531270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:22.531272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:22.531282 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:22.531297 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:22.531304 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:22.531357 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.531436 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.531481 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:10:22.534289 451 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:22.537192 451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:22.551519 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.545455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:45371 (local address 127.0.0.1:52818)
0810 23:10:22.545777 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.545822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.546098 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.547184 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.547192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.547216 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:22.547432 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.547439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.548896 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.548899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.550087 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.550097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.551078 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.551081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.551178 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551203 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.551255 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.551368 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":95,"thread_start_us":99,"threads_started":1}
I20260810 23:10:22.551524 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.546275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.14.75.3:45343)
0810 23:10:22.546752 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.546765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.546788 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:10:22.546867 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.546872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.547048 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.547229 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:22.547559 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.548460 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.550173 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.550831 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.550955 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551149 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.551255 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.551357 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":113,"threads_started":1}
I20260810 23:10:22.551970 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.545575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.14.75.2:45371)
0810 23:10:22.546669 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.546689 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.546717 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:10:22.546796 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.546802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.547010 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.547585 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:22.547706 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.548776 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.550224 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.550933 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.551381 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551565 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.551700 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.551802 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":143,"threads_started":1}
I20260810 23:10:22.552343 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.546345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.14.75.1:35241)
0810 23:10:22.546889 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.546900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.547146 (+ 246us) server_negotiation.cc:407] Connection header received
0810 23:10:22.547259 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.547264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.547435 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.547597 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:22.548000 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.549113 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.550515 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.551318 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.551413 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.551896 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.552003 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":131,"threads_started":1}
I20260810 23:10:22.552858 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.546163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:35241 (local address 127.0.0.1:51418)
0810 23:10:22.547019 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.547034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.547155 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.547742 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.547745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.547762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:22.547866 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.547873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.549245 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.549248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.550372 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.550383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.551631 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.551636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.551727 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551748 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.551840 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.551914 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":66,"thread_start_us":110,"threads_started":1}
I20260810 23:10:22.553037 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.545806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45343 (local address 127.0.0.1:34568)
0810 23:10:22.546196 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.546211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.546345 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.547268 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.547271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.547281 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:22.547432 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.547439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.548575 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.548582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.550033 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.550046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.550956 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.550960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.551064 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.551101 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.551229 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.552310 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":48,"thread_start_us":105,"threads_started":1}
I20260810 23:10:22.554958 713 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe590c0552a4eae94f26b54da7118af (DEFAULT_TABLE table=test-table [id=8bc299cc0ad2451299ff9140a46f14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:22.555979 713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe590c0552a4eae94f26b54da7118af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:22.555966 577 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe590c0552a4eae94f26b54da7118af (DEFAULT_TABLE table=test-table [id=8bc299cc0ad2451299ff9140a46f14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:22.555967 849 tablet_service.cc:1467] Processing CreateTablet for tablet 7fe590c0552a4eae94f26b54da7118af (DEFAULT_TABLE table=test-table [id=8bc299cc0ad2451299ff9140a46f14b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:22.557014 849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe590c0552a4eae94f26b54da7118af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:22.557014 577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fe590c0552a4eae94f26b54da7118af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:22.563004 938 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Bootstrap starting.
I20260810 23:10:22.564935 940 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Bootstrap starting.
I20260810 23:10:22.565490 938 tablet_bootstrap.cc:654] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:22.565742 939 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Bootstrap starting.
I20260810 23:10:22.566424 938 log.cc:824] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:22.567106 940 tablet_bootstrap.cc:654] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:22.567829 939 tablet_bootstrap.cc:654] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:22.567929 940 log.cc:824] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:22.568575 939 log.cc:824] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:22.569190 938 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: No bootstrap required, opened a new log
I20260810 23:10:22.569434 938 ts_tablet_manager.cc:1397] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 23:10:22.569986 940 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: No bootstrap required, opened a new log
I20260810 23:10:22.570168 940 ts_tablet_manager.cc:1397] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:22.570672 939 tablet_bootstrap.cc:492] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: No bootstrap required, opened a new log
I20260810 23:10:22.570865 939 ts_tablet_manager.cc:1397] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:22.572870 938 raft_consensus.cc:348] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.573113 938 raft_consensus.cc:374] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:22.573167 938 raft_consensus.cc:729] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa26e75c128a4e099186de722afb889c, State: Initialized, Role: FOLLOWER
I20260810 23:10:22.573632 938 consensus_queue.cc:260] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.574476 940 raft_consensus.cc:348] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.574735 940 raft_consensus.cc:374] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:22.574793 940 raft_consensus.cc:729] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc96d085772b43f7a0136ab23c4f8d5b, State: Initialized, Role: FOLLOWER
I20260810 23:10:22.575063 938 ts_tablet_manager.cc:1428] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:10:22.575039 939 raft_consensus.cc:348] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.575238 940 consensus_queue.cc:260] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.575627 782 heartbeater.cc:503] Master 127.14.75.62:43779 was elected leader, sending a full tablet report...
I20260810 23:10:22.576188 940 ts_tablet_manager.cc:1428] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:10:22.576653 920 heartbeater.cc:503] Master 127.14.75.62:43779 was elected leader, sending a full tablet report...
I20260810 23:10:22.577471 939 raft_consensus.cc:374] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:22.577561 939 raft_consensus.cc:729] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752ac632422846cdb1318fbb1037c3f3, State: Initialized, Role: FOLLOWER
I20260810 23:10:22.578019 939 consensus_queue.cc:260] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.578779 647 heartbeater.cc:503] Master 127.14.75.62:43779 was elected leader, sending a full tablet report...
I20260810 23:10:22.579025 939 ts_tablet_manager.cc:1428] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260810 23:10:22.595561 648 tablet.cc:2369] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:10:22.757380 921 tablet.cc:2369] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:10:22.816109 783 tablet.cc:2369] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:10:22.967702 944 raft_consensus.cc:482] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:22.967933 944 raft_consensus.cc:504] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.969065 944 leader_election.cc:288] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 752ac632422846cdb1318fbb1037c3f3 (127.14.75.1:35241), bc96d085772b43f7a0136ab23c4f8d5b (127.14.75.3:45343)
I20260810 23:10:22.973284 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.969621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:42339 (local address 127.14.75.3:45343)
0810 23:10:22.969759 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.969763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.969890 (+ 127us) server_negotiation.cc:407] Connection header received
0810 23:10:22.970088 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.970091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.970140 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.970224 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:22.970547 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.971114 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972267 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.972694 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972805 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.972873 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.973072 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.973148 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:10:22.973320 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.969349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:35241 (local address 127.14.75.2:53289)
0810 23:10:22.969971 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.970009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.970113 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.970423 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.970426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.970440 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:22.970595 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.970602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.971358 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.971361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.972129 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.972146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.972860 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.972966 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.972984 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.973081 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.973157 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":179,"thread_start_us":100,"threads_started":1}
I20260810 23:10:22.973321 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.969464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:53289 (local address 127.14.75.1:35241)
0810 23:10:22.969628 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.969640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.970083 (+ 443us) server_negotiation.cc:407] Connection header received
0810 23:10:22.970255 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.970258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.970316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.970398 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:22.970717 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.971237 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972311 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.972700 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972827 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.972885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:22.973121 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:10:22.973200 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:10:22.973335 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.969343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:45343 (local address 127.14.75.2:42339)
0810 23:10:22.969803 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:22.969820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:22.969943 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:22.970254 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:22.970257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:22.970277 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:22.970423 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.970430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.971235 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.971238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.972127 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:22.972136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:22.972835 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:22.972838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:22.972930 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:22.972962 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:10:22.973025 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:10:22.973123 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":1,"thread_start_us":151,"threads_started":1}
I20260810 23:10:22.974114 598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe590c0552a4eae94f26b54da7118af" candidate_uuid: "fa26e75c128a4e099186de722afb889c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752ac632422846cdb1318fbb1037c3f3" is_pre_election: true
I20260810 23:10:22.974190 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe590c0552a4eae94f26b54da7118af" candidate_uuid: "fa26e75c128a4e099186de722afb889c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" is_pre_election: true
I20260810 23:10:22.974435 598 raft_consensus.cc:2454] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa26e75c128a4e099186de722afb889c in term 0.
I20260810 23:10:22.974467 869 raft_consensus.cc:2454] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa26e75c128a4e099186de722afb889c in term 0.
I20260810 23:10:22.975021 666 leader_election.cc:302] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 752ac632422846cdb1318fbb1037c3f3, bc96d085772b43f7a0136ab23c4f8d5b, fa26e75c128a4e099186de722afb889c; no voters:
I20260810 23:10:22.975325 944 raft_consensus.cc:2781] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:22.975415 944 raft_consensus.cc:482] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:22.975458 944 raft_consensus.cc:3037] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:22.976505 944 raft_consensus.cc:504] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.976873 944 leader_election.cc:288] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [CANDIDATE]: Term 1 election: Requested vote from peers 752ac632422846cdb1318fbb1037c3f3 (127.14.75.1:35241), bc96d085772b43f7a0136ab23c4f8d5b (127.14.75.3:45343)
I20260810 23:10:22.977203 598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe590c0552a4eae94f26b54da7118af" candidate_uuid: "fa26e75c128a4e099186de722afb889c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752ac632422846cdb1318fbb1037c3f3"
I20260810 23:10:22.977288 869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fe590c0552a4eae94f26b54da7118af" candidate_uuid: "fa26e75c128a4e099186de722afb889c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc96d085772b43f7a0136ab23c4f8d5b"
I20260810 23:10:22.977303 598 raft_consensus.cc:3037] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:22.977448 869 raft_consensus.cc:3037] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:22.978399 598 raft_consensus.cc:2454] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa26e75c128a4e099186de722afb889c in term 1.
I20260810 23:10:22.978570 869 raft_consensus.cc:2454] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa26e75c128a4e099186de722afb889c in term 1.
I20260810 23:10:22.978713 665 leader_election.cc:302] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752ac632422846cdb1318fbb1037c3f3, fa26e75c128a4e099186de722afb889c; no voters:
I20260810 23:10:22.978927 944 raft_consensus.cc:2781] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:22.979162 944 raft_consensus.cc:686] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 1 LEADER]: Becoming Leader. State: Replica: fa26e75c128a4e099186de722afb889c, State: Running, Role: LEADER
I20260810 23:10:22.979480 944 consensus_queue.cc:237] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } }
I20260810 23:10:22.982805 450 catalog_manager.cc:5495] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c reported cstate change: term changed from 0 to 1, leader changed from <none> to fa26e75c128a4e099186de722afb889c (127.14.75.2). New cstate: current_term: 1 leader_uuid: "fa26e75c128a4e099186de722afb889c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa26e75c128a4e099186de722afb889c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:23.002677 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:22.999030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.14.75.2:45371)
0810 23:10:22.999179 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:22.999184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:22.999200 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:10:22.999332 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:22.999335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:22.999394 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:22.999503 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:22.999784 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:23.000290 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:23.001157 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:23.001377 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:23.001528 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:23.001851 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:23.002260 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:23.002410 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:23.002473 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:10:23.002528 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:10:23.008641 598 raft_consensus.cc:1261] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3 [term 1 FOLLOWER]: Refusing update from remote peer fa26e75c128a4e099186de722afb889c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:23.008814 869 raft_consensus.cc:1261] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 1 FOLLOWER]: Refusing update from remote peer fa26e75c128a4e099186de722afb889c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:23.009451 944 consensus_queue.cc:1035] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752ac632422846cdb1318fbb1037c3f3" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:23.009660 950 consensus_queue.cc:1035] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc96d085772b43f7a0136ab23c4f8d5b" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:23.026592 451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:23.026800 451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:23.029203 451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cb50e7f264440c2ae96377490ecb7ed: Sending DeleteTablet for 3 replicas of tablet 7fe590c0552a4eae94f26b54da7118af
I20260810 23:10:23.029959 713 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe590c0552a4eae94f26b54da7118af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:23 UTC) from {username='slave'} at 127.0.0.1:52818
I20260810 23:10:23.030086 577 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe590c0552a4eae94f26b54da7118af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:23 UTC) from {username='slave'} at 127.0.0.1:51418
I20260810 23:10:23.030143 849 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fe590c0552a4eae94f26b54da7118af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:23 UTC) from {username='slave'} at 127.0.0.1:34568
I20260810 23:10:23.030483 963 tablet_replica.cc:330] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: stopping tablet replica
I20260810 23:10:23.030596 964 tablet_replica.cc:330] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: stopping tablet replica
I20260810 23:10:23.030821 965 tablet_replica.cc:330] T 7fe590c0552a4eae94f26b54da7118af P 752ac632422846cdb1318fbb1037c3f3: stopping tablet replica
I20260810 23:10:23.031179 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 512
I20260810 23:10:23.031188 964 raft_consensus.cc:2229] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:23.031533 964 raft_consensus.cc:2258] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:23.033700 964 ts_tablet_manager.cc:1905] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:23.037878 964 ts_tablet_manager.cc:1918] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:23.038095 964 log.cc:1196] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/7fe590c0552a4eae94f26b54da7118af
I20260810 23:10:23.038612 964 ts_tablet_manager.cc:1939] T 7fe590c0552a4eae94f26b54da7118af P bc96d085772b43f7a0136ab23c4f8d5b: Deleting consensus metadata
I20260810 23:10:23.040146 435 catalog_manager.cc:4841] TS bc96d085772b43f7a0136ab23c4f8d5b (127.14.75.3:45343): tablet 7fe590c0552a4eae94f26b54da7118af (table test-table [id=8bc299cc0ad2451299ff9140a46f14b7]) successfully deleted
I20260810 23:10:23.040470 963 raft_consensus.cc:2229] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:10:23.040913 963 raft_consensus.cc:2258] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:23.042645 963 ts_tablet_manager.cc:1905] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:10:23.044987 434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:35241 (error 108)
I20260810 23:10:23.046073 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:23.045759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:35241 (local address 127.0.0.1:51424)
0810 23:10:23.045911 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:23.045984 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:35241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:10:23.046186 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 650
I20260810 23:10:23.046288 963 ts_tablet_manager.cc:1918] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:23.046386 963 log.cc:1196] T 7fe590c0552a4eae94f26b54da7118af P fa26e75c128a4e099186de722afb889c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/7fe590c0552a4eae94f26b54da7118af
W20260810 23:10:23.046348 434 catalog_manager.cc:4584] TS 752ac632422846cdb1318fbb1037c3f3 (127.14.75.1:35241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fe590c0552a4eae94f26b54da7118af: Network error: Client connection negotiation failed: client connection to 127.14.75.1:35241: connect: Connection refused (error 111)
I20260810 23:10:23.053139 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 785
I20260810 23:10:23.055022 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:23.053391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:45371 (local address 127.0.0.1:52836)
0810 23:10:23.054838 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:23.054910 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.2:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1330}
I20260810 23:10:23.059863 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 418
2026-08-10T23:10:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1436 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:23 dist-test-slave-srgv krb5kdc[970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:23 dist-test-slave-srgv krb5kdc[970](info): set up 2 sockets
Aug 10 23:10:23 dist-test-slave-srgv krb5kdc[970](info): commencing operation
krb5kdc: starting...
W20260810 23:10:24.129987 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:24.255954 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41569
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:24.367180 983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:24.367470 983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:24.367511 983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:24.371464 983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:24.371526 983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:24.371546 983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:24.371562 983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:24.371577 983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:24.376842 983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:24.378005 983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:24.378973 983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:24.387411 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.387459 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.387415 988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:24.387698 983 server_base.cc:1034] running on GCE node
I20260810 23:10:24.388424 983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:24.389628 983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:24.390812 983 hybrid_clock.cc:648] HybridClock initialized: now 1786403424390802 us; error 31 us; skew 500 ppm
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:24.399763 983 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:24.400740 983 webserver.cc:460] Webserver started at http://127.14.75.62:41023/ using document root <none> and password file <none>
I20260810 23:10:24.401351 983 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:24.401481 983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:24.401721 983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:24.403905 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "4e8b1c936ec64bd5abd674997105cad4"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "e0f9ba38e230db7ae680cefc708221cb"
server_key_iv: "fdf104e95af3d888b21f37992398383d"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.404528 983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "4e8b1c936ec64bd5abd674997105cad4"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "e0f9ba38e230db7ae680cefc708221cb"
server_key_iv: "fdf104e95af3d888b21f37992398383d"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.408457 983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:10:24.410975 999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.412242 983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:24.412392 983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "4e8b1c936ec64bd5abd674997105cad4"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "e0f9ba38e230db7ae680cefc708221cb"
server_key_iv: "fdf104e95af3d888b21f37992398383d"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.412515 983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:24.429347 983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:24.432227 983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:24.432389 983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:24.439786 983 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41569
I20260810 23:10:24.439802 1053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41569 every 8 connection(s)
I20260810 23:10:24.440732 983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:24.443331 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 983
I20260810 23:10:24.443477 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:24.443650 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:24.443786 14636 external_mini_cluster.cc:1428] Setting key cad39012c81af150ccaae4d65aa80be1
I20260810 23:10:24.450222 1054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: Bootstrap starting.
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:24.453233 1054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:24.454257 1054 log.cc:824] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:24.456777 1054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: No bootstrap required, opened a new log
I20260810 23:10:24.459079 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:24.445252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.14.75.62:41569)
0810 23:10:24.445758 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:24.445772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:24.445808 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:10:24.446438 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:24.446454 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:24.446772 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:24.447108 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:24.447941 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.448948 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.449893 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.450366 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.450544 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:24.453558 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:24.453567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:24.453587 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:24.453624 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:24.455565 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.456034 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.456040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.456045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.456188 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.456460 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.456474 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.456475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.457022 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:24.457223 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:24.457754 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 23:10:24.458069 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":187,"threads_started":1}
I20260810 23:10:24.461488 1054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:10:24.461764 1054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:24.461807 1054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8b1c936ec64bd5abd674997105cad4, State: Initialized, Role: FOLLOWER
I20260810 23:10:24.462219 1054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:10:24.462340 1054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:24.462387 1054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:24.462457 1054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:24.463368 1054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:10:24.463670 1054 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8b1c936ec64bd5abd674997105cad4; no voters:
I20260810 23:10:24.463937 1054 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:24.464089 1059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:24.464334 1059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [term 1 LEADER]: Becoming Leader. State: Replica: 4e8b1c936ec64bd5abd674997105cad4, State: Running, Role: LEADER
I20260810 23:10:24.464778 1059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } }
I20260810 23:10:24.465354 1054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:24.466440 1060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e8b1c936ec64bd5abd674997105cad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } } }
I20260810 23:10:24.466570 1060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:24.467015 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8b1c936ec64bd5abd674997105cad4. Latest consensus state: current_term: 1 leader_uuid: "4e8b1c936ec64bd5abd674997105cad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8b1c936ec64bd5abd674997105cad4" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41569 } } }
I20260810 23:10:24.467768 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4 [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:24.470165 1075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:24.470234 1075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:24.470314 1068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:24.471019 1068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:24.476722 1068 catalog_manager.cc:1347] Generated new cluster ID: ccb311f13a6f474e9dc6f438e9c45128
I20260810 23:10:24.476809 1068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:24.495872 1068 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:24.497221 1068 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:24.505393 1068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: Generated new TSK 0
I20260810 23:10:24.506331 1068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:10:24.548174 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:24.662634 1080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:24.662964 1080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:24.663043 1080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:24.667573 1080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:24.667663 1080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:24.667775 1080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:24.673141 1080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:24.674300 1080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:24.675349 1080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:24.684967 1088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.684979 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.684976 1086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:24.685524 1080 server_base.cc:1034] running on GCE node
I20260810 23:10:24.685986 1080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:24.686653 1080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:24.687848 1080 hybrid_clock.cc:648] HybridClock initialized: now 1786403424687833 us; error 28 us; skew 500 ppm
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:24.690840 1080 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:10:24.691731 1080 webserver.cc:460] Webserver started at http://127.14.75.1:40607/ using document root <none> and password file <none>
I20260810 23:10:24.692307 1080 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:24.692431 1080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:24.692698 1080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:24.694880 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "14daab290a1a43a89715d76512c69f18"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "9dfaca3f30d21e47d9e4f98798bdbde6"
server_key_iv: "e69e87f5a1b680ee2340f4a7281159db"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.695475 1080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "14daab290a1a43a89715d76512c69f18"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "9dfaca3f30d21e47d9e4f98798bdbde6"
server_key_iv: "e69e87f5a1b680ee2340f4a7281159db"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.699287 1080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:10:24.701761 1095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.702965 1080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:10:24.703104 1080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "14daab290a1a43a89715d76512c69f18"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "9dfaca3f30d21e47d9e4f98798bdbde6"
server_key_iv: "e69e87f5a1b680ee2340f4a7281159db"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.703222 1080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:24.718173 1080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:24.721661 1080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:24.721882 1080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:24.722496 1080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:24.723342 1080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:24.723418 1080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.723491 1080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:24.723534 1080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.732236 1080 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:34195
I20260810 23:10:24.732272 1214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:34195 every 8 connection(s)
I20260810 23:10:24.733121 1080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:24.735749 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1080
I20260810 23:10:24.735944 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:24.736244 14636 external_mini_cluster.cc:1428] Setting key b7d0e0151af8346df3ced3adb29797cc
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:10:24.748174 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:24.734933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:35537 (local address 127.14.75.62:41569)
0810 23:10:24.735102 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:24.735107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:24.735973 (+ 866us) server_negotiation.cc:407] Connection header received
0810 23:10:24.737572 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:24.737575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:24.737627 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:24.737757 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:24.739343 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.739983 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.741137 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.741596 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.741722 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:24.744282 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:24.744286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:24.744289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:24.744320 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:24.746056 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.746575 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.746579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.746581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.746683 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.747018 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.747024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.747025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.747293 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:24.747413 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:24.747878 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 23:10:24.748012 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:10:24.749101 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:24.735310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.1:35537)
0810 23:10:24.735803 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:24.735857 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:24.737315 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:24.737956 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:24.737965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:24.738473 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:24.739174 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:24.739196 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.740126 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.740129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:24.740994 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:24.741004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.741750 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.741752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:24.741890 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:24.742584 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:24.742606 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:24.744094 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:24.746191 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:24.746195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:24.746207 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:24.746462 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:24.746787 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:24.746789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:24.746790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:24.746918 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:24.747421 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:24.747426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:24.747655 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 23:10:24.747824 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 23:10:24.748199 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
I20260810 23:10:24.750483 1215 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
I20260810 23:10:24.750751 1215 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:24.751328 1215 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:10:24.752946 1017 ts_manager.cc:194] Registered new tserver with Master: 14daab290a1a43a89715d76512c69f18 (127.14.75.1:34195)
I20260810 23:10:24.754236 1017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:35537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:10:24.768155 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:24.883213 1220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:24.883468 1220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:24.883513 1220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:24.887849 1220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:24.887912 1220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:24.887974 1220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:24.893361 1220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:24.894596 1220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:24.895609 1220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:24.905346 1226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.905346 1229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:24.905391 1227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:24.905876 1220 server_base.cc:1034] running on GCE node
I20260810 23:10:24.906430 1220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:24.907192 1220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:24.908376 1220 hybrid_clock.cc:648] HybridClock initialized: now 1786403424908343 us; error 47 us; skew 500 ppm
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:24.911345 1220 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:10:24.912211 1220 webserver.cc:460] Webserver started at http://127.14.75.2:45581/ using document root <none> and password file <none>
I20260810 23:10:24.912735 1220 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:24.912803 1220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:24.912973 1220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:24.915050 1220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "ebf9c4c38c4847479fcaa3a8d263e342"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "85c6c691711843a0dc561836ec065edb"
server_key_iv: "324f65a21c0833ceef864d2a763a76a6"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.915647 1220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "ebf9c4c38c4847479fcaa3a8d263e342"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "85c6c691711843a0dc561836ec065edb"
server_key_iv: "324f65a21c0833ceef864d2a763a76a6"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.919541 1220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:10:24.922099 1237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.923307 1220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:24.923404 1220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "ebf9c4c38c4847479fcaa3a8d263e342"
format_stamp: "Formatted at 2026-08-10 23:10:24 on dist-test-slave-srgv"
server_key: "85c6c691711843a0dc561836ec065edb"
server_key_iv: "324f65a21c0833ceef864d2a763a76a6"
server_key_version: "encryptionkey@0"
I20260810 23:10:24.923493 1220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:24.934955 1220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:24.937958 1220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:24.938128 1220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:24.938656 1220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:24.939466 1220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:24.939515 1220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.939555 1220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:24.939579 1220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:24.948730 1220 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:46523
I20260810 23:10:24.948761 1356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:46523 every 8 connection(s)
I20260810 23:10:24.949779 1220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:24.955379 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1220
I20260810 23:10:24.955525 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:24.955832 14636 external_mini_cluster.cc:1428] Setting key afececbb5b32698af67c321cc62c74f1
Aug 10 23:10:24 dist-test-slave-srgv krb5kdc[970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403424, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:24.964200 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:24.951534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:56481 (local address 127.14.75.62:41569)
0810 23:10:24.951687 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:24.951690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:24.952487 (+ 797us) server_negotiation.cc:407] Connection header received
0810 23:10:24.953801 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:24.953804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:24.953851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:24.953928 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:24.955131 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.955963 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.957284 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.957596 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.957724 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:24.960317 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:24.960327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:24.960329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:24.960352 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:24.961958 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.962530 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.962533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.962535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.962623 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:24.963004 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:24.963009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:24.963011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:24.963222 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:24.963338 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:24.963878 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 23:10:24.964016 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:10:24.965168 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:24.951883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.2:56481)
0810 23:10:24.952326 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:24.952370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:24.953600 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:24.954091 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:24.954098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:24.954483 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:24.954978 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:24.954991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.956091 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.956094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:24.957141 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:24.957153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:24.957746 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:24.957750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:24.957904 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:24.958562 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:24.958579 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:24.960125 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:24.962097 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:24.962104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:24.962121 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:24.962416 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:24.962725 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:24.962729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:24.962731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:24.962901 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:24.963340 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:24.963349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:24.963624 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 23:10:24.963833 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 23:10:24.964208 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260810 23:10:24.966588 1357 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:10:24.966862 1357 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:24.967522 1357 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:10:24.968746 1017 ts_manager.cc:194] Registered new tserver with Master: ebf9c4c38c4847479fcaa3a8d263e342 (127.14.75.2:46523)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
I20260810 23:10:24.969589 1017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:56481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:10:24.984823 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:25.099341 1362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:25.099629 1362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:25.099673 1362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:25.103916 1362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:25.103989 1362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:25.104068 1362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:25.109723 1362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:41569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:25.110824 1362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:25.111851 1362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:25.121948 1367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.122058 1362 server_base.cc:1034] running on GCE node
W20260810 23:10:25.121939 1368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.121968 1370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.122603 1362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:25.123301 1362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:25.124498 1362 hybrid_clock.cc:648] HybridClock initialized: now 1786403425124472 us; error 49 us; skew 500 ppm
Aug 10 23:10:25 dist-test-slave-srgv krb5kdc[970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403425, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:25.127525 1362 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:10:25.128409 1362 webserver.cc:460] Webserver started at http://127.14.75.3:44137/ using document root <none> and password file <none>
I20260810 23:10:25.128933 1362 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:25.128993 1362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:25.129158 1362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:25.131376 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "c646b4a6c99844d3b1ca995811652d61"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "dbc1ce0b9a7ed5d0db3e3cc91280dcdf"
server_key_iv: "550eb52ca1aeb2453f2c8117a1c6ba6c"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.131999 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "c646b4a6c99844d3b1ca995811652d61"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "dbc1ce0b9a7ed5d0db3e3cc91280dcdf"
server_key_iv: "550eb52ca1aeb2453f2c8117a1c6ba6c"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.135751 1362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:10:25.138185 1377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.139429 1362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:25.139528 1362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "c646b4a6c99844d3b1ca995811652d61"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "dbc1ce0b9a7ed5d0db3e3cc91280dcdf"
server_key_iv: "550eb52ca1aeb2453f2c8117a1c6ba6c"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.139618 1362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:25.151286 1362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:25.154901 1362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:25.155079 1362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:25.155748 1362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:25.156651 1362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:25.156700 1362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.156780 1362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:25.156819 1362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.166142 1362 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:37675
I20260810 23:10:25.166204 1495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:37675 every 8 connection(s)
I20260810 23:10:25.167251 1362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:25.172066 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1362
I20260810 23:10:25.172214 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:25.172513 14636 external_mini_cluster.cc:1428] Setting key f1ebe421b054fffaf11416e338aaf6f5
Aug 10 23:10:25 dist-test-slave-srgv krb5kdc[970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403425, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:25.181383 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.169123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:55581 (local address 127.14.75.62:41569)
0810 23:10:25.169276 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.169280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.170124 (+ 844us) server_negotiation.cc:407] Connection header received
0810 23:10:25.171543 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.171547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.171604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.171720 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:25.173211 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.173814 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.174996 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.175226 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.175344 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.177878 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:25.177886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:25.177888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:25.177912 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:25.179239 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:25.179787 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:25.179792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:25.179794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:25.179877 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:25.180232 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:25.180235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:25.180237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:25.180426 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:25.180540 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.181042 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.181172 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:10:25.182209 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.169514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41569 (local address 127.14.75.3:55581)
0810 23:10:25.169970 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.170014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.171317 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.171919 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.171926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.172414 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:25.173051 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.173079 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.173960 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.173964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.174854 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.174863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.175387 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.175391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.175559 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.176124 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:25.176140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:25.177708 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:25.179374 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:25.179380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:25.179401 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:25.179672 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:25.179975 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:25.179979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:25.179981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:25.180132 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:25.180538 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:25.180545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:25.180801 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.180999 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.181348 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260810 23:10:25.183437 1497 heartbeater.cc:347] Connected to a master server at 127.14.75.62:41569
I20260810 23:10:25.183686 1497 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:25.184201 1497 heartbeater.cc:511] Master 127.14.75.62:41569 requested a full tablet report, sending...
I20260810 23:10:25.185272 1017 ts_manager.cc:194] Registered new tserver with Master: c646b4a6c99844d3b1ca995811652d61 (127.14.75.3:37675)
I20260810 23:10:25.185972 1017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:55581
I20260810 23:10:25.186534 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:25.196488 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.188633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.14.75.62:41569)
0810 23:10:25.188727 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.188760 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.188975 (+ 215us) server_negotiation.cc:407] Connection header received
0810 23:10:25.189115 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.189118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.189159 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.189251 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:25.190043 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.190551 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.191444 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.191673 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.191798 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.193371 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:25.193375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:25.193378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:25.193420 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:25.194585 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:25.195105 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:25.195109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:25.195111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:25.195197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:25.195542 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:25.195546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:25.195547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:25.195773 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:25.195939 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.196214 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.196314 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:10:25.199501 1016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:25.202674 1016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:25.217752 1511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.211359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:34195 (local address 127.0.0.1:53862)
0810 23:10:25.212302 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.212317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.212400 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.213475 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.213478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.213497 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:25.213594 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.213600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.214960 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.214965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.216131 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.216140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.217121 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.217124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.217255 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.217321 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.217452 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.217575 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":24,"thread_start_us":108,"threads_started":1}
I20260810 23:10:25.217969 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.211412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51336 (local address 127.14.75.3:37675)
0810 23:10:25.212238 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.212265 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.212289 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:10:25.212372 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.212378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.212577 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.213231 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:25.213365 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.214607 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.216372 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.217232 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.217432 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.217575 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.217693 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.217791 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":184,"threads_started":1}
I20260810 23:10:25.218529 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.212085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.14.75.1:34195)
0810 23:10:25.212981 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.212997 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.213029 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:10:25.213132 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.213137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.213298 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.213489 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:25.213740 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.214832 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.216264 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.217012 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.217121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.217267 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.218277 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.218362 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":95,"threads_started":1}
I20260810 23:10:25.218654 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.212495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.14.75.2:46523)
0810 23:10:25.213815 (+ 1320us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.213824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.213842 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:10:25.213902 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.213905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.214025 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.214173 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:25.214407 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.215493 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.216909 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.217808 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.217908 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.218034 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.218368 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.218481 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":128,"threads_started":1}
I20260810 23:10:25.217962 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.211268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:37675 (local address 127.0.0.1:51336)
0810 23:10:25.211740 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.211801 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.211907 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.212767 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.212773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.212795 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:25.213022 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.213035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.214752 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.214755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.216154 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.216165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.217438 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.217442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.217544 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.217568 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.217652 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.217723 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":9,"thread_start_us":91,"threads_started":1}
I20260810 23:10:25.218520 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.212046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:46523 (local address 127.0.0.1:34622)
0810 23:10:25.212779 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.212793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.213023 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.214154 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.214157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.214175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:25.214297 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.214307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.215644 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.215647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.216796 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.216807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.218104 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.218110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.218215 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.218232 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.218333 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.218402 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":107,"threads_started":1}
I20260810 23:10:25.221586 1288 tablet_service.cc:1467] Processing CreateTablet for tablet 0245cbe97a8d47bea86a11b5882af752 (DEFAULT_TABLE table=test-table [id=d17fed00cb184933873f2cc4369c34e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:25.221704 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 0245cbe97a8d47bea86a11b5882af752 (DEFAULT_TABLE table=test-table [id=d17fed00cb184933873f2cc4369c34e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:25.221704 1147 tablet_service.cc:1467] Processing CreateTablet for tablet 0245cbe97a8d47bea86a11b5882af752 (DEFAULT_TABLE table=test-table [id=d17fed00cb184933873f2cc4369c34e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:25.222712 1288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0245cbe97a8d47bea86a11b5882af752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:25.222735 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0245cbe97a8d47bea86a11b5882af752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:25.222728 1147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0245cbe97a8d47bea86a11b5882af752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:25.229256 1514 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Bootstrap starting.
I20260810 23:10:25.230616 1515 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: Bootstrap starting.
I20260810 23:10:25.232139 1514 tablet_bootstrap.cc:654] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:25.232424 1515 tablet_bootstrap.cc:654] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:25.233340 1514 log.cc:824] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:25.233485 1515 log.cc:824] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:25.233877 1516 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Bootstrap starting.
I20260810 23:10:25.235690 1514 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: No bootstrap required, opened a new log
I20260810 23:10:25.235690 1515 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: No bootstrap required, opened a new log
I20260810 23:10:25.235898 1514 ts_tablet_manager.cc:1397] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 23:10:25.235922 1515 ts_tablet_manager.cc:1397] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:25.235929 1516 tablet_bootstrap.cc:654] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:25.237284 1516 log.cc:824] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:25.239145 1516 tablet_bootstrap.cc:492] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: No bootstrap required, opened a new log
I20260810 23:10:25.239365 1516 ts_tablet_manager.cc:1397] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:10:25.240046 1514 raft_consensus.cc:348] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.240046 1515 raft_consensus.cc:348] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.240334 1514 raft_consensus.cc:374] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:25.240334 1515 raft_consensus.cc:374] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:25.240396 1514 raft_consensus.cc:729] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf9c4c38c4847479fcaa3a8d263e342, State: Initialized, Role: FOLLOWER
I20260810 23:10:25.240396 1515 raft_consensus.cc:729] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14daab290a1a43a89715d76512c69f18, State: Initialized, Role: FOLLOWER
I20260810 23:10:25.240875 1515 consensus_queue.cc:260] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.240875 1514 consensus_queue.cc:260] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.241799 1357 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:10:25.241953 1515 ts_tablet_manager.cc:1428] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:10:25.241953 1514 ts_tablet_manager.cc:1428] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 23:10:25.243065 1215 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:10:25.244201 1516 raft_consensus.cc:348] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.244571 1516 raft_consensus.cc:374] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:25.244642 1516 raft_consensus.cc:729] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c646b4a6c99844d3b1ca995811652d61, State: Initialized, Role: FOLLOWER
I20260810 23:10:25.245129 1516 consensus_queue.cc:260] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.246017 1497 heartbeater.cc:503] Master 127.14.75.62:41569 was elected leader, sending a full tablet report...
I20260810 23:10:25.246291 1516 ts_tablet_manager.cc:1428] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260810 23:10:25.259788 1521 raft_consensus.cc:482] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:25.259974 1521 raft_consensus.cc:504] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.261037 1521 leader_election.cc:288] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c646b4a6c99844d3b1ca995811652d61 (127.14.75.3:37675), ebf9c4c38c4847479fcaa3a8d263e342 (127.14.75.2:46523)
I20260810 23:10:25.265714 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.261365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:59645 (local address 127.14.75.3:37675)
0810 23:10:25.261505 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.261509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.261650 (+ 141us) server_negotiation.cc:407] Connection header received
0810 23:10:25.261729 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.261734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.261794 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.261884 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:25.262261 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.262937 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.264521 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.265149 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.265258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.265334 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.265505 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.265583 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:10:25.265969 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.261843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:55921 (local address 127.14.75.2:46523)
0810 23:10:25.261975 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.261981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.262071 (+ 90us) server_negotiation.cc:407] Connection header received
0810 23:10:25.262156 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.262158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.262213 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.262297 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:25.262563 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.263298 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.264708 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.265324 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.265454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.265520 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.265717 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.265809 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:10:25.266010 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.261243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:37675 (local address 127.14.75.1:59645)
0810 23:10:25.261509 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.261526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.261640 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.261974 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.261978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.262002 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:25.262144 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.262154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.263039 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.263046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.264391 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.264402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.265268 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.265271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.265347 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.265368 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.265487 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.265579 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260810 23:10:25.266201 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.261645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:46523 (local address 127.14.75.1:55921)
0810 23:10:25.261976 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.261988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.262075 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.262317 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.262320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.262336 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:25.262442 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.262446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.263446 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.263449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.264596 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.264604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.265474 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.265480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.265579 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.265600 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.265674 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.265733 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":156,"threads_started":1}
I20260810 23:10:25.266629 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0245cbe97a8d47bea86a11b5882af752" candidate_uuid: "14daab290a1a43a89715d76512c69f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c646b4a6c99844d3b1ca995811652d61" is_pre_election: true
I20260810 23:10:25.266955 1448 raft_consensus.cc:2454] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14daab290a1a43a89715d76512c69f18 in term 0.
I20260810 23:10:25.267503 1099 leader_election.cc:302] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14daab290a1a43a89715d76512c69f18, c646b4a6c99844d3b1ca995811652d61; no voters:
I20260810 23:10:25.267736 1309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0245cbe97a8d47bea86a11b5882af752" candidate_uuid: "14daab290a1a43a89715d76512c69f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" is_pre_election: true
I20260810 23:10:25.267820 1521 raft_consensus.cc:2781] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:25.267927 1521 raft_consensus.cc:482] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:25.268006 1521 raft_consensus.cc:3037] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:25.268029 1309 raft_consensus.cc:2454] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14daab290a1a43a89715d76512c69f18 in term 0.
I20260810 23:10:25.269184 1521 raft_consensus.cc:504] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.269582 1521 leader_election.cc:288] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [CANDIDATE]: Term 1 election: Requested vote from peers c646b4a6c99844d3b1ca995811652d61 (127.14.75.3:37675), ebf9c4c38c4847479fcaa3a8d263e342 (127.14.75.2:46523)
I20260810 23:10:25.270006 1448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0245cbe97a8d47bea86a11b5882af752" candidate_uuid: "14daab290a1a43a89715d76512c69f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c646b4a6c99844d3b1ca995811652d61"
I20260810 23:10:25.270061 1309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0245cbe97a8d47bea86a11b5882af752" candidate_uuid: "14daab290a1a43a89715d76512c69f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf9c4c38c4847479fcaa3a8d263e342"
I20260810 23:10:25.270161 1448 raft_consensus.cc:3037] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:25.270182 1309 raft_consensus.cc:3037] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:25.271234 1448 raft_consensus.cc:2454] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14daab290a1a43a89715d76512c69f18 in term 1.
I20260810 23:10:25.271260 1309 raft_consensus.cc:2454] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14daab290a1a43a89715d76512c69f18 in term 1.
I20260810 23:10:25.271572 1099 leader_election.cc:302] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14daab290a1a43a89715d76512c69f18, ebf9c4c38c4847479fcaa3a8d263e342; no voters:
I20260810 23:10:25.271759 1521 raft_consensus.cc:2781] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:25.272061 1521 raft_consensus.cc:686] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 1 LEADER]: Becoming Leader. State: Replica: 14daab290a1a43a89715d76512c69f18, State: Running, Role: LEADER
I20260810 23:10:25.272428 1521 consensus_queue.cc:237] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } }
I20260810 23:10:25.275554 1017 catalog_manager.cc:5495] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14daab290a1a43a89715d76512c69f18 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "14daab290a1a43a89715d76512c69f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14daab290a1a43a89715d76512c69f18" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 34195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:25.300441 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.296667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.14.75.1:34195)
0810 23:10:25.296813 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.296820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.296836 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:10:25.297047 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.297050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.297114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.297239 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:25.297550 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.298060 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.298932 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.299150 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.299259 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.299586 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:25.299995 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:25.300159 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.300248 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.300296 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:10:25.307253 1448 raft_consensus.cc:1261] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 1 FOLLOWER]: Refusing update from remote peer 14daab290a1a43a89715d76512c69f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:25.307216 1309 raft_consensus.cc:1261] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 1 FOLLOWER]: Refusing update from remote peer 14daab290a1a43a89715d76512c69f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:25.307988 1521 consensus_queue.cc:1035] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c646b4a6c99844d3b1ca995811652d61" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 37675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:25.308153 1526 consensus_queue.cc:1035] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf9c4c38c4847479fcaa3a8d263e342" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:25.323211 1017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:25.323387 1017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:25.325675 1017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e8b1c936ec64bd5abd674997105cad4: Sending DeleteTablet for 3 replicas of tablet 0245cbe97a8d47bea86a11b5882af752
I20260810 23:10:25.326502 1288 tablet_service.cc:1514] Processing DeleteTablet for tablet 0245cbe97a8d47bea86a11b5882af752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:25 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:34622
I20260810 23:10:25.326570 1147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0245cbe97a8d47bea86a11b5882af752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:25 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:53862
I20260810 23:10:25.327190 1538 tablet_replica.cc:330] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18: stopping tablet replica
I20260810 23:10:25.327214 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1080
I20260810 23:10:25.327162 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 0245cbe97a8d47bea86a11b5882af752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:25 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:51336
I20260810 23:10:25.327620 1538 raft_consensus.cc:2229] T 0245cbe97a8d47bea86a11b5882af752 P 14daab290a1a43a89715d76512c69f18 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:10:25.328048 1539 tablet_replica.cc:330] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: stopping tablet replica
I20260810 23:10:25.328337 1539 raft_consensus.cc:2229] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:25.328701 1539 raft_consensus.cc:2258] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:25.329258 1540 tablet_replica.cc:330] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: stopping tablet replica
I20260810 23:10:25.329617 1540 raft_consensus.cc:2229] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:25.329931 1540 raft_consensus.cc:2258] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:25.331115 1539 ts_tablet_manager.cc:1905] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:25.332476 1540 ts_tablet_manager.cc:1905] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:25.333640 1539 ts_tablet_manager.cc:1918] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:25.333757 1539 log.cc:1196] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/0245cbe97a8d47bea86a11b5882af752
I20260810 23:10:25.334125 1539 ts_tablet_manager.cc:1939] T 0245cbe97a8d47bea86a11b5882af752 P ebf9c4c38c4847479fcaa3a8d263e342: Deleting consensus metadata
I20260810 23:10:25.335345 1003 catalog_manager.cc:4841] TS ebf9c4c38c4847479fcaa3a8d263e342 (127.14.75.2:46523): tablet 0245cbe97a8d47bea86a11b5882af752 (table test-table [id=d17fed00cb184933873f2cc4369c34e0]) successfully deleted
I20260810 23:10:25.335937 1540 ts_tablet_manager.cc:1918] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:25.336051 1540 log.cc:1196] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/0245cbe97a8d47bea86a11b5882af752
W20260810 23:10:25.336148 1000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:34195 (error 108)
I20260810 23:10:25.336404 1540 ts_tablet_manager.cc:1939] T 0245cbe97a8d47bea86a11b5882af752 P c646b4a6c99844d3b1ca995811652d61: Deleting consensus metadata
I20260810 23:10:25.337041 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1220
I20260810 23:10:25.337257 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.336954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:34195 (local address 127.0.0.1:53868)
0810 23:10:25.337099 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.337170 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:34195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:10:25.339298 1003 catalog_manager.cc:4841] TS c646b4a6c99844d3b1ca995811652d61 (127.14.75.3:37675): tablet 0245cbe97a8d47bea86a11b5882af752 (table test-table [id=d17fed00cb184933873f2cc4369c34e0]) successfully deleted
W20260810 23:10:25.341619 1000 catalog_manager.cc:4584] TS 14daab290a1a43a89715d76512c69f18 (127.14.75.1:34195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0245cbe97a8d47bea86a11b5882af752: Network error: Client connection negotiation failed: client connection to 127.14.75.1:34195: connect: Connection refused (error 111)
I20260810 23:10:25.343732 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1362
I20260810 23:10:25.349936 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 983
2026-08-10T23:10:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2290 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:25Z Disabled control of system clock
I20260810 23:10:25.381671 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:34543
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:25.497112 1545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:25.497673 1545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:25.497766 1545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:25.501848 1545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:25.501919 1545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:25.501933 1545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:25.501951 1545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:25.501966 1545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:25.506981 1545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:34543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:25.508085 1545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:25.509534 1545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:25.518350 1554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.518321 1551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.518364 1552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.518661 1545 server_base.cc:1034] running on GCE node
I20260810 23:10:25.519398 1545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:25.520574 1545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:25.521761 1545 hybrid_clock.cc:648] HybridClock initialized: now 1786403425521748 us; error 29 us; skew 500 ppm
I20260810 23:10:25.523618 1545 webserver.cc:460] Webserver started at http://127.14.75.62:42171/ using document root <none> and password file <none>
I20260810 23:10:25.524237 1545 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:25.524335 1545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:25.524573 1545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:25.526703 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "7fa3657be6d44473ae8297b5792ad979"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "32e0d30f4d4d8170f1a0942f8bf4f2ce"
server_key_iv: "79298f292dd7ed7dab7ad366848110b2"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.527305 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "7fa3657be6d44473ae8297b5792ad979"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "32e0d30f4d4d8170f1a0942f8bf4f2ce"
server_key_iv: "79298f292dd7ed7dab7ad366848110b2"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.531055 1545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:10:25.533473 1560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.534708 1545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:25.534876 1545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "7fa3657be6d44473ae8297b5792ad979"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "32e0d30f4d4d8170f1a0942f8bf4f2ce"
server_key_iv: "79298f292dd7ed7dab7ad366848110b2"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.535004 1545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:25.547469 1545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:25.548148 1545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:25.548368 1545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:25.555572 1545 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:34543
I20260810 23:10:25.555596 1619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:34543 every 8 connection(s)
I20260810 23:10:25.556734 1545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:25.557847 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1545
I20260810 23:10:25.557987 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:25.558336 14636 external_mini_cluster.cc:1428] Setting key 18caf9256767ab5adb8abe05a1ded8e4
I20260810 23:10:25.560693 1620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:25.566632 1620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: Bootstrap starting.
I20260810 23:10:25.568383 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.559858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.14.75.62:34543)
0810 23:10:25.560547 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.560562 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.560603 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:10:25.561329 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.561346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.561827 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.562197 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:25.562653 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.563645 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.564584 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.565071 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.565275 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.566251 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:25.566256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:25.566269 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:25.566698 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:25.566792 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:25.566823 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:25.566930 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.567143 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.567424 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":178,"threads_started":1}
I20260810 23:10:25.569516 1620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:25.570436 1620 log.cc:824] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:25.572456 1620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: No bootstrap required, opened a new log
I20260810 23:10:25.576084 1620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } }
I20260810 23:10:25.576327 1620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:25.576402 1620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa3657be6d44473ae8297b5792ad979, State: Initialized, Role: FOLLOWER
I20260810 23:10:25.576931 1620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } }
I20260810 23:10:25.577076 1620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:25.577121 1620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:25.577193 1620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:25.578187 1620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } }
I20260810 23:10:25.578488 1620 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa3657be6d44473ae8297b5792ad979; no voters:
I20260810 23:10:25.578769 1620 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:25.578934 1625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:25.579186 1625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa3657be6d44473ae8297b5792ad979, State: Running, Role: LEADER
I20260810 23:10:25.579532 1625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } }
I20260810 23:10:25.580058 1620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:25.581174 1627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa3657be6d44473ae8297b5792ad979. Latest consensus state: current_term: 1 leader_uuid: "7fa3657be6d44473ae8297b5792ad979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } } }
I20260810 23:10:25.581214 1626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa3657be6d44473ae8297b5792ad979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa3657be6d44473ae8297b5792ad979" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 34543 } } }
I20260810 23:10:25.581337 1627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:25.581337 1626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:25.581877 1634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:25.585234 1634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:25.591045 1634 catalog_manager.cc:1347] Generated new cluster ID: 96982d8bf37246bf91f3d7d7d8f787b1
I20260810 23:10:25.591125 1634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:25.617364 1634 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:25.618363 1634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:25.626433 1634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: Generated new TSK 0
I20260810 23:10:25.627151 1634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:25.629768 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--builtin_ntp_servers=127.14.75.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:25.744444 1645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:25.744699 1645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:25.744738 1645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:25.748771 1645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:25.748828 1645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:25.748883 1645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:25.753574 1645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:25.754544 1645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:25.755913 1645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:25.765470 1654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.765532 1652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.765532 1651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.765843 1645 server_base.cc:1034] running on GCE node
I20260810 23:10:25.766304 1645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:25.766937 1645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:25.768126 1645 hybrid_clock.cc:648] HybridClock initialized: now 1786403425768087 us; error 51 us; skew 500 ppm
I20260810 23:10:25.770069 1645 webserver.cc:460] Webserver started at http://127.14.75.1:45517/ using document root <none> and password file <none>
I20260810 23:10:25.770671 1645 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:25.770777 1645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:25.771006 1645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:25.773137 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "f6934f8052ba47fe9a7885633ae1cb5d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "9fd7cc655b365a4d8b50bbab068d66ab"
server_key_iv: "a7956cbbdfc19a41ced1c767c414c377"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.773787 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "f6934f8052ba47fe9a7885633ae1cb5d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "9fd7cc655b365a4d8b50bbab068d66ab"
server_key_iv: "a7956cbbdfc19a41ced1c767c414c377"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.777624 1645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:10:25.780041 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.781486 1645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:25.781607 1645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "f6934f8052ba47fe9a7885633ae1cb5d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "9fd7cc655b365a4d8b50bbab068d66ab"
server_key_iv: "a7956cbbdfc19a41ced1c767c414c377"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.781736 1645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:25.806213 1645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:25.806986 1645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:25.807199 1645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:25.807822 1645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:25.808802 1645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:25.808876 1645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.808946 1645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:25.808995 1645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.818163 1645 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:44357
I20260810 23:10:25.818216 1781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:44357 every 8 connection(s)
I20260810 23:10:25.819077 1645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:25.826943 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1645
I20260810 23:10:25.827099 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:25.827428 14636 external_mini_cluster.cc:1428] Setting key b5fde64f711c7067a17a91812ca74c81
I20260810 23:10:25.828413 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.820909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:34719 (local address 127.14.75.62:34543)
0810 23:10:25.821050 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:25.821054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:25.822061 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 23:10:25.823448 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:25.823452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:25.823514 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:25.823594 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:25.824484 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.825018 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.826215 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.826458 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.826575 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.827794 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:25.827801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:25.827804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:25.827820 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:25.827843 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:25.827854 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:25.827932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:25.828162 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:10:25.828276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:10:25.829531 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:25.821291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:34543 (local address 127.14.75.1:34719)
0810 23:10:25.821906 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:25.821945 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:25.823234 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:25.823763 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:25.823770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:25.823850 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:25.824335 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.824347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.825152 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.825156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.826082 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:25.826091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:25.826618 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:25.826622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:25.826790 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:25.827484 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:25.827486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:25.827552 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:25.827635 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:25.827956 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:25.827988 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:25.827997 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:10:25.828162 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 23:10:25.828428 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":117,"threads_started":1}
I20260810 23:10:25.830152 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--builtin_ntp_servers=127.14.75.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:25.830868 1782 heartbeater.cc:347] Connected to a master server at 127.14.75.62:34543
I20260810 23:10:25.831130 1782 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:25.831775 1782 heartbeater.cc:511] Master 127.14.75.62:34543 requested a full tablet report, sending...
I20260810 23:10:25.833678 1580 ts_manager.cc:194] Registered new tserver with Master: f6934f8052ba47fe9a7885633ae1cb5d (127.14.75.1:44357)
I20260810 23:10:25.835407 1580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.1:34719
W20260810 23:10:25.943806 1785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:25.944239 1785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:25.944336 1785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:25.951103 1785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:25.951208 1785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:25.951324 1785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:25.956593 1785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:25.957718 1785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:25.959056 1785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:25.968904 1792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:25.968971 1793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.969151 1785 server_base.cc:1034] running on GCE node
W20260810 23:10:25.968912 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:25.969679 1785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:25.970355 1785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:25.971540 1785 hybrid_clock.cc:648] HybridClock initialized: now 1786403425971529 us; error 29 us; skew 500 ppm
I20260810 23:10:25.973273 1785 webserver.cc:460] Webserver started at http://127.14.75.2:34513/ using document root <none> and password file <none>
I20260810 23:10:25.973898 1785 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:25.973995 1785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:25.974210 1785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:25.976320 1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "e7280b35a21f4cbba766c749774e5a2d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "4c897cab52e06931a845be6b3c601fca"
server_key_iv: "d7cf922ee096d5e35ef4a11f696d7a47"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.976922 1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "e7280b35a21f4cbba766c749774e5a2d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "4c897cab52e06931a845be6b3c601fca"
server_key_iv: "d7cf922ee096d5e35ef4a11f696d7a47"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.980733 1785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:10:25.983209 1802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:25.984417 1785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:10:25.984568 1785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "e7280b35a21f4cbba766c749774e5a2d"
format_stamp: "Formatted at 2026-08-10 23:10:25 on dist-test-slave-srgv"
server_key: "4c897cab52e06931a845be6b3c601fca"
server_key_iv: "d7cf922ee096d5e35ef4a11f696d7a47"
server_key_version: "encryptionkey@0"
I20260810 23:10:25.984730 1785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:26.015019 1785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:26.015796 1785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:26.016023 1785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:26.016649 1785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:26.017655 1785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:26.017733 1785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:26.017794 1785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:26.017848 1785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:26.026957 1785 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:34351
I20260810 23:10:26.026984 1916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:34351 every 8 connection(s)
I20260810 23:10:26.027859 1785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:26.028694 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1785
I20260810 23:10:26.028820 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:26.029170 14636 external_mini_cluster.cc:1428] Setting key 66a3568178ca431b826f9441164a35e0
I20260810 23:10:26.031278 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--builtin_ntp_servers=127.14.75.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:26.038787 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.029766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:33943 (local address 127.14.75.62:34543)
0810 23:10:26.030113 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.030118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.031043 (+ 925us) server_negotiation.cc:407] Connection header received
0810 23:10:26.032576 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.032580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.032637 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.032717 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:26.034014 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.034541 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.036377 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.036596 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.036686 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.038203 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:26.038206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:26.038208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:26.038222 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:26.038242 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:26.038256 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:26.038316 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.038527 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.038585 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260810 23:10:26.040094 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.030174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:34543 (local address 127.14.75.2:33943)
0810 23:10:26.030893 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.030946 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.032333 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.032902 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.032914 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.033046 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:26.033831 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.033853 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.034682 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.034689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.036244 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.036261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.036797 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.036804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.037058 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.037856 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:26.037861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:26.037930 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:26.038029 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:26.038351 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:26.038364 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:26.038372 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.038529 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.038902 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260810 23:10:26.041395 1917 heartbeater.cc:347] Connected to a master server at 127.14.75.62:34543
I20260810 23:10:26.041729 1917 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:26.042302 1917 heartbeater.cc:511] Master 127.14.75.62:34543 requested a full tablet report, sending...
I20260810 23:10:26.043434 1580 ts_manager.cc:194] Registered new tserver with Master: e7280b35a21f4cbba766c749774e5a2d (127.14.75.2:34351)
I20260810 23:10:26.044014 1580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.2:33943
W20260810 23:10:26.148342 1920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:26.148667 1920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:26.148737 1920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:26.152771 1920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:26.152879 1920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:26.152977 1920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:26.157862 1920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:34543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:26.159044 1920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:26.160472 1920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:26.170368 1930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:26.170361 1926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:26.170359 1927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:26.170675 1920 server_base.cc:1034] running on GCE node
I20260810 23:10:26.171166 1920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:26.171819 1920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:26.172976 1920 hybrid_clock.cc:648] HybridClock initialized: now 1786403426172940 us; error 47 us; skew 500 ppm
I20260810 23:10:26.174892 1920 webserver.cc:460] Webserver started at http://127.14.75.3:35131/ using document root <none> and password file <none>
I20260810 23:10:26.175482 1920 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:26.175545 1920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:26.175793 1920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:26.177983 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "c6de4bff88f4400faf8d1602d80ada3d"
format_stamp: "Formatted at 2026-08-10 23:10:26 on dist-test-slave-srgv"
server_key: "851ddc11897a6dcb4fe05d352e6d094a"
server_key_iv: "6f7a9d9bdeb71350fac02eee9fd6f99d"
server_key_version: "encryptionkey@0"
I20260810 23:10:26.178589 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "c6de4bff88f4400faf8d1602d80ada3d"
format_stamp: "Formatted at 2026-08-10 23:10:26 on dist-test-slave-srgv"
server_key: "851ddc11897a6dcb4fe05d352e6d094a"
server_key_iv: "6f7a9d9bdeb71350fac02eee9fd6f99d"
server_key_version: "encryptionkey@0"
I20260810 23:10:26.182487 1920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:10:26.184849 1936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:26.186223 1920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:26.186352 1920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "c6de4bff88f4400faf8d1602d80ada3d"
format_stamp: "Formatted at 2026-08-10 23:10:26 on dist-test-slave-srgv"
server_key: "851ddc11897a6dcb4fe05d352e6d094a"
server_key_iv: "6f7a9d9bdeb71350fac02eee9fd6f99d"
server_key_version: "encryptionkey@0"
I20260810 23:10:26.186482 1920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:26.200681 1920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:26.201363 1920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:26.201586 1920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:26.202173 1920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:26.203111 1920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:26.203183 1920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:26.203254 1920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:26.203290 1920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:26.212111 1920 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:41577
I20260810 23:10:26.212141 2050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:41577 every 8 connection(s)
I20260810 23:10:26.213003 1920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:26.219115 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 1920
I20260810 23:10:26.219254 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:26.219537 14636 external_mini_cluster.cc:1428] Setting key af37f63ba35047e165ca771f04472360
I20260810 23:10:26.222709 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.214809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:59833 (local address 127.14.75.62:34543)
0810 23:10:26.214957 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.214961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.216004 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 23:10:26.217436 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.217440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.217508 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.217617 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:26.218514 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.219048 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.220595 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.220927 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.221041 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.222313 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:26.222315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:26.222317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:26.222330 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:26.222349 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:26.222356 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:26.222419 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.222551 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.222601 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:10:26.223603 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.215179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:34543 (local address 127.14.75.3:59833)
0810 23:10:26.215845 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.215886 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.217189 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.217802 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.217809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.217890 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:26.218357 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.218369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.219194 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.219201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.220469 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.220482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.221109 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.221112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.221307 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.222064 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:26.222066 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:26.222105 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:26.222168 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:26.222432 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:26.222439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:26.222443 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.222536 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.222751 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260810 23:10:26.224702 2051 heartbeater.cc:347] Connected to a master server at 127.14.75.62:34543
I20260810 23:10:26.224951 2051 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:26.225498 2051 heartbeater.cc:511] Master 127.14.75.62:34543 requested a full tablet report, sending...
I20260810 23:10:26.226511 1580 ts_manager.cc:194] Registered new tserver with Master: c6de4bff88f4400faf8d1602d80ada3d (127.14.75.3:41577)
I20260810 23:10:26.227082 1580 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.3:59833
I20260810 23:10:26.233626 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:26.239396 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.235616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.14.75.62:34543)
0810 23:10:26.235754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.235757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.235938 (+ 181us) server_negotiation.cc:407] Connection header received
0810 23:10:26.236024 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.236027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.236066 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.236132 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:26.236400 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.236897 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.237754 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.237965 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.238043 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.238975 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:26.238977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:26.238980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:26.238991 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:26.239008 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:26.239015 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:26.239114 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.239249 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.239287 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:10:26.242429 1580 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:26.245471 1580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:26.259759 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.254871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.14.75.1:44357)
0810 23:10:26.255324 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.255339 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.255364 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:10:26.255443 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.255448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.255649 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.255805 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:26.255976 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.257090 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.258498 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259055 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259185 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.259393 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.259472 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.259569 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":114,"threads_started":1}
I20260810 23:10:26.259754 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.254418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:44357 (local address 127.0.0.1:52302)
0810 23:10:26.254675 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.254689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.254774 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.255790 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.255792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.255809 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:26.255881 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.255886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.257215 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.257218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.258378 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.258386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259186 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.259295 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.259332 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.259444 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.259588 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":55,"thread_start_us":106,"threads_started":1}
I20260810 23:10:26.260316 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.253868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.14.75.3:41577)
0810 23:10:26.254834 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.254848 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.254878 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:10:26.254977 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.254982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.255190 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.255590 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:26.255844 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.257090 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.258577 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259358 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259833 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.260009 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.260060 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.260145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":165,"threads_started":1}
I20260810 23:10:26.260317 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.254694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.14.75.2:34351)
0810 23:10:26.255082 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.255092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.255114 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:10:26.255181 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.255185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.255355 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.255830 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:26.255975 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.256692 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.258332 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259122 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259833 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.260009 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.260060 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.260144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":188,"threads_started":1}
I20260810 23:10:26.259754 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.254032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:34351 (local address 127.0.0.1:37070)
0810 23:10:26.254457 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.254471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.254573 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.255491 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.255493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.255506 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:26.255678 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.255686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.256827 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.256830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.258186 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.258196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259279 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.259372 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.259412 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.259467 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.259583 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":64,"thread_start_us":115,"threads_started":1}
I20260810 23:10:26.260847 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.253657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:41577 (local address 127.0.0.1:41444)
0810 23:10:26.254010 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.254063 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.254193 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.255378 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.255383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.255409 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:26.255686 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.255700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.257257 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.257260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.258415 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.258425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.259492 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.259496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.259574 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.259593 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.259642 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.259701 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":101,"thread_start_us":105,"threads_started":1}
I20260810 23:10:26.263123 1711 tablet_service.cc:1467] Processing CreateTablet for tablet 933f70bab3c14535b91c6310ccd2d532 (DEFAULT_TABLE table=test-table [id=b0a102a8d7524b28982ff2e944096d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:26.263222 1850 tablet_service.cc:1467] Processing CreateTablet for tablet 933f70bab3c14535b91c6310ccd2d532 (DEFAULT_TABLE table=test-table [id=b0a102a8d7524b28982ff2e944096d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:26.264125 1850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933f70bab3c14535b91c6310ccd2d532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:26.264127 1711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933f70bab3c14535b91c6310ccd2d532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:26.267226 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 933f70bab3c14535b91c6310ccd2d532 (DEFAULT_TABLE table=test-table [id=b0a102a8d7524b28982ff2e944096d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:26.268205 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933f70bab3c14535b91c6310ccd2d532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:26.270488 2069 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Bootstrap starting.
I20260810 23:10:26.272490 2069 tablet_bootstrap.cc:654] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:26.273283 2069 log.cc:824] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:26.274641 2071 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Bootstrap starting.
I20260810 23:10:26.275465 2072 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Bootstrap starting.
I20260810 23:10:26.275558 2069 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: No bootstrap required, opened a new log
I20260810 23:10:26.275766 2069 ts_tablet_manager.cc:1397] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:26.277089 2071 tablet_bootstrap.cc:654] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:26.277745 2072 tablet_bootstrap.cc:654] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:26.277963 2071 log.cc:824] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:26.278560 2072 log.cc:824] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:26.280208 2071 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: No bootstrap required, opened a new log
I20260810 23:10:26.280339 2072 tablet_bootstrap.cc:492] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: No bootstrap required, opened a new log
I20260810 23:10:26.280150 2069 raft_consensus.cc:348] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.280407 2071 ts_tablet_manager.cc:1397] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:10:26.280408 2069 raft_consensus.cc:374] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:26.280467 2069 raft_consensus.cc:729] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7280b35a21f4cbba766c749774e5a2d, State: Initialized, Role: FOLLOWER
I20260810 23:10:26.280524 2072 ts_tablet_manager.cc:1397] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:26.280973 2069 consensus_queue.cc:260] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.281720 1917 heartbeater.cc:503] Master 127.14.75.62:34543 was elected leader, sending a full tablet report...
I20260810 23:10:26.282015 2069 ts_tablet_manager.cc:1428] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:10:26.284623 2072 raft_consensus.cc:348] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.284911 2072 raft_consensus.cc:374] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:26.284749 2071 raft_consensus.cc:348] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.284996 2072 raft_consensus.cc:729] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6de4bff88f4400faf8d1602d80ada3d, State: Initialized, Role: FOLLOWER
I20260810 23:10:26.284993 2071 raft_consensus.cc:374] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:26.285051 2071 raft_consensus.cc:729] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6934f8052ba47fe9a7885633ae1cb5d, State: Initialized, Role: FOLLOWER
I20260810 23:10:26.285501 2072 consensus_queue.cc:260] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.285581 2071 consensus_queue.cc:260] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.286177 1782 heartbeater.cc:503] Master 127.14.75.62:34543 was elected leader, sending a full tablet report...
I20260810 23:10:26.286540 2072 ts_tablet_manager.cc:1428] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:10:26.286547 2071 ts_tablet_manager.cc:1428] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:10:26.286872 2051 heartbeater.cc:503] Master 127.14.75.62:34543 was elected leader, sending a full tablet report...
W20260810 23:10:26.320297 1783 tablet.cc:2369] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:10:26.339905 2077 raft_consensus.cc:482] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:26.340102 2077 raft_consensus.cc:504] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.341254 2077 leader_election.cc:288] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6de4bff88f4400faf8d1602d80ada3d (127.14.75.3:41577), e7280b35a21f4cbba766c749774e5a2d (127.14.75.2:34351)
I20260810 23:10:26.346690 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.341691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:56133 (local address 127.14.75.2:34351)
0810 23:10:26.341838 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.341842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.342038 (+ 196us) server_negotiation.cc:407] Connection header received
0810 23:10:26.342124 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.342127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.342182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.342270 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:26.342727 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.343513 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.345360 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.345989 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.346105 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.346167 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.346455 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.346567 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:10:26.346690 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.341624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:40975 (local address 127.14.75.3:41577)
0810 23:10:26.341777 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.341782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.341972 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:10:26.342126 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.342128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.342180 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.342278 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:26.342727 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.343513 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.345360 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.345989 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.346105 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.346167 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.346452 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.346554 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:26.346730 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.341486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:41577 (local address 127.14.75.1:40975)
0810 23:10:26.341859 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.341875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.342002 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.342308 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.342311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.342353 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:26.342609 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.342615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.343636 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.343640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.345224 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.345236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.346124 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.346128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.346293 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.346332 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.346416 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.346536 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":360,"spinlock_wait_cycles":7040,"thread_start_us":114,"threads_started":1}
I20260810 23:10:26.346741 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.341597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:34351 (local address 127.14.75.1:56133)
0810 23:10:26.341947 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.341960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:26.342044 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:26.342308 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:26.342311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:26.342352 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:26.342599 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.342608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.343636 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.343640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.345223 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:26.345236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.346124 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.346128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:26.346288 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.346342 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:10:26.346416 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:10:26.346536 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":217,"spinlock_wait_cycles":29696,"thread_start_us":110,"threads_started":1}
I20260810 23:10:26.347448 2005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933f70bab3c14535b91c6310ccd2d532" candidate_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6de4bff88f4400faf8d1602d80ada3d" is_pre_election: true
I20260810 23:10:26.347620 1871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933f70bab3c14535b91c6310ccd2d532" candidate_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7280b35a21f4cbba766c749774e5a2d" is_pre_election: true
I20260810 23:10:26.347805 2005 raft_consensus.cc:2454] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6934f8052ba47fe9a7885633ae1cb5d in term 0.
I20260810 23:10:26.347996 1871 raft_consensus.cc:2454] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6934f8052ba47fe9a7885633ae1cb5d in term 0.
I20260810 23:10:26.348443 1665 leader_election.cc:302] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c6de4bff88f4400faf8d1602d80ada3d, e7280b35a21f4cbba766c749774e5a2d, f6934f8052ba47fe9a7885633ae1cb5d; no voters:
I20260810 23:10:26.348755 2077 raft_consensus.cc:2781] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:26.348845 2077 raft_consensus.cc:482] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:26.348888 2077 raft_consensus.cc:3037] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:26.350051 2077 raft_consensus.cc:504] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.350453 2077 leader_election.cc:288] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [CANDIDATE]: Term 1 election: Requested vote from peers c6de4bff88f4400faf8d1602d80ada3d (127.14.75.3:41577), e7280b35a21f4cbba766c749774e5a2d (127.14.75.2:34351)
I20260810 23:10:26.350771 2005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933f70bab3c14535b91c6310ccd2d532" candidate_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6de4bff88f4400faf8d1602d80ada3d"
I20260810 23:10:26.350831 1871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933f70bab3c14535b91c6310ccd2d532" candidate_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7280b35a21f4cbba766c749774e5a2d"
I20260810 23:10:26.350896 2005 raft_consensus.cc:3037] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:26.350947 1871 raft_consensus.cc:3037] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:26.352182 2005 raft_consensus.cc:2454] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6934f8052ba47fe9a7885633ae1cb5d in term 1.
I20260810 23:10:26.352182 1871 raft_consensus.cc:2454] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6934f8052ba47fe9a7885633ae1cb5d in term 1.
I20260810 23:10:26.352523 1665 leader_election.cc:302] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6de4bff88f4400faf8d1602d80ada3d, f6934f8052ba47fe9a7885633ae1cb5d; no voters:
I20260810 23:10:26.352730 2077 raft_consensus.cc:2781] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:26.352921 2077 raft_consensus.cc:686] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [term 1 LEADER]: Becoming Leader. State: Replica: f6934f8052ba47fe9a7885633ae1cb5d, State: Running, Role: LEADER
I20260810 23:10:26.353235 2077 consensus_queue.cc:237] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } }
I20260810 23:10:26.356357 1579 catalog_manager.cc:5495] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d reported cstate change: term changed from 0 to 1, leader changed from <none> to f6934f8052ba47fe9a7885633ae1cb5d (127.14.75.1). New cstate: current_term: 1 leader_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6934f8052ba47fe9a7885633ae1cb5d" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 44357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:26.383358 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.379816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.14.75.1:44357)
0810 23:10:26.379980 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:26.379983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:26.379999 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:10:26.380068 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:26.380071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:26.380124 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:26.380210 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:26.380439 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.380998 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.381978 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:26.382177 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:26.382261 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:26.382539 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:26.382908 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:26.383013 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:26.383119 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:10:26.383178 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:10:26.389241 1871 raft_consensus.cc:1261] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 1 FOLLOWER]: Refusing update from remote peer f6934f8052ba47fe9a7885633ae1cb5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:26.389241 2005 raft_consensus.cc:1261] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 1 FOLLOWER]: Refusing update from remote peer f6934f8052ba47fe9a7885633ae1cb5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:26.389972 2077 consensus_queue.cc:1035] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6de4bff88f4400faf8d1602d80ada3d" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:26.390165 2080 consensus_queue.cc:1035] T 933f70bab3c14535b91c6310ccd2d532 P f6934f8052ba47fe9a7885633ae1cb5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7280b35a21f4cbba766c749774e5a2d" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:26.400333 2085 mvcc.cc:205] Tried to move back new op lower bound from 7317108434482798592 to 7317108434345500672. Current Snapshot: MvccSnapshot[applied={T|T < 7317108434482798592 or (T in {7317108434482798592})}]
I20260810 23:10:26.405864 1580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:26.406050 1580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:26.408578 1580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fa3657be6d44473ae8297b5792ad979: Sending DeleteTablet for 3 replicas of tablet 933f70bab3c14535b91c6310ccd2d532
I20260810 23:10:26.409466 1850 tablet_service.cc:1514] Processing DeleteTablet for tablet 933f70bab3c14535b91c6310ccd2d532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:26 UTC) from {username='slave'} at 127.0.0.1:37070
I20260810 23:10:26.409582 1985 tablet_service.cc:1514] Processing DeleteTablet for tablet 933f70bab3c14535b91c6310ccd2d532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:26 UTC) from {username='slave'} at 127.0.0.1:41444
I20260810 23:10:26.410032 2095 tablet_replica.cc:330] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: stopping tablet replica
I20260810 23:10:26.410118 2096 tablet_replica.cc:330] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: stopping tablet replica
I20260810 23:10:26.410439 2095 raft_consensus.cc:2229] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:26.410437 2096 raft_consensus.cc:2229] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:26.410636 1711 tablet_service.cc:1514] Processing DeleteTablet for tablet 933f70bab3c14535b91c6310ccd2d532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:26 UTC) from {username='slave'} at 127.0.0.1:52302
I20260810 23:10:26.410766 2096 raft_consensus.cc:2258] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:26.410758 2095 raft_consensus.cc:2258] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:26.411149 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1645
I20260810 23:10:26.413336 2096 ts_tablet_manager.cc:1905] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:26.413336 2095 ts_tablet_manager.cc:1905] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:26.416145 2096 ts_tablet_manager.cc:1918] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:26.416145 2095 ts_tablet_manager.cc:1918] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:26.416260 2096 log.cc:1196] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/933f70bab3c14535b91c6310ccd2d532
I20260810 23:10:26.416266 2095 log.cc:1196] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/933f70bab3c14535b91c6310ccd2d532
I20260810 23:10:26.416592 2096 ts_tablet_manager.cc:1939] T 933f70bab3c14535b91c6310ccd2d532 P c6de4bff88f4400faf8d1602d80ada3d: Deleting consensus metadata
I20260810 23:10:26.416592 2095 ts_tablet_manager.cc:1939] T 933f70bab3c14535b91c6310ccd2d532 P e7280b35a21f4cbba766c749774e5a2d: Deleting consensus metadata
I20260810 23:10:26.417971 1563 catalog_manager.cc:4841] TS e7280b35a21f4cbba766c749774e5a2d (127.14.75.2:34351): tablet 933f70bab3c14535b91c6310ccd2d532 (table test-table [id=b0a102a8d7524b28982ff2e944096d19]) successfully deleted
I20260810 23:10:26.418015 1565 catalog_manager.cc:4841] TS c6de4bff88f4400faf8d1602d80ada3d (127.14.75.3:41577): tablet 933f70bab3c14535b91c6310ccd2d532 (table test-table [id=b0a102a8d7524b28982ff2e944096d19]) successfully deleted
W20260810 23:10:26.419173 1565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:44357 (error 108)
I20260810 23:10:26.419435 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1785
I20260810 23:10:26.421679 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:26.421275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:44357 (local address 127.0.0.1:54212)
0810 23:10:26.421484 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:26.421572 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 23:10:26.421836 1565 catalog_manager.cc:4584] TS f6934f8052ba47fe9a7885633ae1cb5d (127.14.75.1:44357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 933f70bab3c14535b91c6310ccd2d532: Network error: Client connection negotiation failed: client connection to 127.14.75.1:44357: connect: Connection refused (error 111)
I20260810 23:10:26.426142 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1920
I20260810 23:10:26.432446 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 1545
2026-08-10T23:10:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1081 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:26 dist-test-slave-srgv krb5kdc[2101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:26 dist-test-slave-srgv krb5kdc[2101](info): set up 2 sockets
Aug 10 23:10:26 dist-test-slave-srgv krb5kdc[2101](info): commencing operation
krb5kdc: starting...
W20260810 23:10:27.493942 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:27 dist-test-slave-srgv krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:27.618160 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40331
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:40331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:27.728732 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:27.728993 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:27.729034 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:27.732926 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:27.732986 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:27.733006 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:27.733021 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:27.733036 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:27.738170 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:40331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:40331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:27.739123 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:27.740070 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:27.748282 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:27.748288 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:27.748379 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:27.748358 2115 server_base.cc:1034] running on GCE node
I20260810 23:10:27.749114 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:27.750241 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:27.751407 2115 hybrid_clock.cc:648] HybridClock initialized: now 1786403427751378 us; error 44 us; skew 500 ppm
Aug 10 23:10:27 dist-test-slave-srgv krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403427, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:27.754287 2115 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:27.755139 2115 webserver.cc:460] Webserver started at http://127.14.75.62:41069/ using document root <none> and password file <none>
I20260810 23:10:27.755687 2115 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:27.755797 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:27.756027 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:27.758111 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "b601b46e073b4433b094ac0d1a8d6f95"
format_stamp: "Formatted at 2026-08-10 23:10:27 on dist-test-slave-srgv"
server_key: "5f2950e4123d13fc6ab017285dd79a50"
server_key_iv: "8b8a51be850cdb3dd25de2c07917c05d"
server_key_version: "encryptionkey@0"
I20260810 23:10:27.758669 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "b601b46e073b4433b094ac0d1a8d6f95"
format_stamp: "Formatted at 2026-08-10 23:10:27 on dist-test-slave-srgv"
server_key: "5f2950e4123d13fc6ab017285dd79a50"
server_key_iv: "8b8a51be850cdb3dd25de2c07917c05d"
server_key_version: "encryptionkey@0"
I20260810 23:10:27.762359 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:10:27.764719 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:27.765942 2115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:27.766090 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "b601b46e073b4433b094ac0d1a8d6f95"
format_stamp: "Formatted at 2026-08-10 23:10:27 on dist-test-slave-srgv"
server_key: "5f2950e4123d13fc6ab017285dd79a50"
server_key_iv: "8b8a51be850cdb3dd25de2c07917c05d"
server_key_version: "encryptionkey@0"
I20260810 23:10:27.766207 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:27.790714 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:27.794212 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:27.794445 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:27.801620 2115 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:40331
I20260810 23:10:27.801646 2182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:40331 every 8 connection(s)
I20260810 23:10:27.802659 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:27.804431 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2115
I20260810 23:10:27.804584 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:27.804858 14636 external_mini_cluster.cc:1428] Setting key 75037ace381739d6409a3d0277fdb07a
I20260810 23:10:27.805753 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:27.812587 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: Bootstrap starting.
Aug 10 23:10:27 dist-test-slave-srgv krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:27.815136 2183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:27.816118 2183 log.cc:824] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:27.818499 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: No bootstrap required, opened a new log
I20260810 23:10:27.820585 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:27.806303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.14.75.62:40331)
0810 23:10:27.806739 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:27.806758 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:27.806799 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:10:27.807704 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:27.807727 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:27.808140 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:27.808502 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:27.809175 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:27.810727 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:27.811620 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:27.812305 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:27.812466 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:27.815272 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:27.815279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:27.815297 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:27.815331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:27.817397 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:27.817881 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:27.817889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:27.817895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:27.818029 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:27.818354 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:27.818356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:27.818358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:27.818865 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:27.819054 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:27.819324 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:10:27.819624 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260810 23:10:27.822647 2183 raft_consensus.cc:348] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } }
I20260810 23:10:27.822934 2183 raft_consensus.cc:374] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:27.823012 2183 raft_consensus.cc:729] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b601b46e073b4433b094ac0d1a8d6f95, State: Initialized, Role: FOLLOWER
I20260810 23:10:27.823374 2183 consensus_queue.cc:260] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } }
I20260810 23:10:27.823486 2183 raft_consensus.cc:388] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:27.823529 2183 raft_consensus.cc:482] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:27.823606 2183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:27.824537 2183 raft_consensus.cc:504] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } }
I20260810 23:10:27.824828 2183 leader_election.cc:302] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b601b46e073b4433b094ac0d1a8d6f95; no voters:
I20260810 23:10:27.825078 2183 leader_election.cc:288] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:27.825217 2188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:27.825508 2188 raft_consensus.cc:686] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [term 1 LEADER]: Becoming Leader. State: Replica: b601b46e073b4433b094ac0d1a8d6f95, State: Running, Role: LEADER
I20260810 23:10:27.826061 2188 consensus_queue.cc:237] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } }
I20260810 23:10:27.826323 2183 sys_catalog.cc:564] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:27.827764 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b601b46e073b4433b094ac0d1a8d6f95. Latest consensus state: current_term: 1 leader_uuid: "b601b46e073b4433b094ac0d1a8d6f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } } }
I20260810 23:10:27.827795 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b601b46e073b4433b094ac0d1a8d6f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b601b46e073b4433b094ac0d1a8d6f95" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 40331 } } }
I20260810 23:10:27.827905 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:27.827906 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95 [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:27.831014 2203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:27.831086 2203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:27.831171 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:27.831883 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:27.837669 2200 catalog_manager.cc:1347] Generated new cluster ID: 32fabc3992c142e3b57e76cd9801ba32
I20260810 23:10:27.837741 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:27.850384 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:27.851708 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:27.859905 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: Generated new TSK 0
I20260810 23:10:27.860733 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.75.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
Entry for principal kudu/127.14.75.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab.
I20260810 23:10:27.894618 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:28.017868 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:28.018121 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:28.018163 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:28.022209 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:28.022261 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:28.022317 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:28.027596 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.1.keytab
--principal=kudu/127.14.75.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:28.028625 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:28.029636 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:28.038858 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.038882 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.038878 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:28.039268 2209 server_base.cc:1034] running on GCE node
I20260810 23:10:28.039611 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:28.040146 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:28.041306 2209 hybrid_clock.cc:648] HybridClock initialized: now 1786403428041275 us; error 41 us; skew 500 ppm
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:28.044157 2209 init.cc:377] Logged in from keytab as kudu/127.14.75.1@KRBTEST.COM (short username kudu)
I20260810 23:10:28.045042 2209 webserver.cc:460] Webserver started at http://127.14.75.1:37919/ using document root <none> and password file <none>
I20260810 23:10:28.045676 2209 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:28.045742 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:28.045953 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:28.047926 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "003d5e3f29d0449fa1b6bba8f586a721"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "b0fa651f72cf3d8173097d7e537ee56a"
server_key_iv: "9bd66161294457def3646a72b054f33c"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.048496 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "003d5e3f29d0449fa1b6bba8f586a721"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "b0fa651f72cf3d8173097d7e537ee56a"
server_key_iv: "9bd66161294457def3646a72b054f33c"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.052409 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:10:28.054955 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.056190 2209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:28.056331 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "003d5e3f29d0449fa1b6bba8f586a721"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "b0fa651f72cf3d8173097d7e537ee56a"
server_key_iv: "9bd66161294457def3646a72b054f33c"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.056449 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:28.078889 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:28.082293 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:28.082496 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:28.083076 2209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:28.083902 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:28.083972 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.084054 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:28.084106 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.093129 2209 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:42137
I20260810 23:10:28.093158 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:42137 every 8 connection(s)
I20260810 23:10:28.094022 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:28.102177 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2209
I20260810 23:10:28.102360 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:28.102690 14636 external_mini_cluster.cc:1428] Setting key 9ad04f3558e517ab592357547954cf40
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.1@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:28.108893 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.095844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:37399 (local address 127.14.75.62:40331)
0810 23:10:28.095991 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.095995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.096805 (+ 810us) server_negotiation.cc:407] Connection header received
0810 23:10:28.098206 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.098208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.098261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.098352 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:28.099787 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.100334 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.101489 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.101724 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.101830 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.104655 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:28.104658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:28.104660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:28.104681 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:28.106618 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.107139 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.107145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.107149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.107248 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.107585 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.107591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.107596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.107863 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:28.107969 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.108452 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.108618 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:10:28.109707 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.096209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40331 (local address 127.14.75.1:37399)
0810 23:10:28.096651 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.096692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.097966 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.098518 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.098525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.099009 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:28.099632 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.099648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.100456 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.100458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.101335 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.101345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.101880 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.101883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.102021 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.102662 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:28.102678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:28.104488 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:28.106756 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.106760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.106778 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.107036 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.107353 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.107355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.107357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.107490 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.107969 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:28.107975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:28.108208 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.108365 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.108644 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":124,"threads_started":1}
I20260810 23:10:28.111006 2344 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40331
I20260810 23:10:28.111289 2344 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:28.111789 2344 heartbeater.cc:511] Master 127.14.75.62:40331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.2@KRBTEST.COM; defaulting to no policy
I20260810 23:10:28.113332 2147 ts_manager.cc:194] Registered new tserver with Master: 003d5e3f29d0449fa1b6bba8f586a721 (127.14.75.1:42137)
I20260810 23:10:28.114641 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.1@KRBTEST.COM'} at 127.14.75.1:37399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
Entry for principal kudu/127.14.75.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab.
I20260810 23:10:28.130790 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:28.240447 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:28.240710 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:28.240788 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:28.244845 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:28.244907 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:28.245023 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:28.250290 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.2.keytab
--principal=kudu/127.14.75.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:28.251281 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:28.252297 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:28.261914 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.261914 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.261909 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:28.262202 2349 server_base.cc:1034] running on GCE node
I20260810 23:10:28.262602 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:28.263258 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:28.264406 2349 hybrid_clock.cc:648] HybridClock initialized: now 1786403428264392 us; error 26 us; skew 500 ppm
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:28.267213 2349 init.cc:377] Logged in from keytab as kudu/127.14.75.2@KRBTEST.COM (short username kudu)
I20260810 23:10:28.268061 2349 webserver.cc:460] Webserver started at http://127.14.75.2:34747/ using document root <none> and password file <none>
I20260810 23:10:28.268604 2349 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:28.268666 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:28.268913 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:28.270995 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "5eac73b216eb43c6887766f426117a45"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "e84375184de466ad9006c720e5590e5b"
server_key_iv: "d899c30d5306999d8722fe6153e5fd14"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.271546 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "5eac73b216eb43c6887766f426117a45"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "e84375184de466ad9006c720e5590e5b"
server_key_iv: "d899c30d5306999d8722fe6153e5fd14"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.275235 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:10:28.277568 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.278718 2349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:28.278873 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "5eac73b216eb43c6887766f426117a45"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "e84375184de466ad9006c720e5590e5b"
server_key_iv: "d899c30d5306999d8722fe6153e5fd14"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.278983 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:28.304934 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:28.308373 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:28.308596 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:28.309216 2349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:28.310117 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:28.310192 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.310259 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:28.310312 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.319211 2349 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:36553
I20260810 23:10:28.319244 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:36553 every 8 connection(s)
I20260810 23:10:28.320111 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:28.327548 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2349
I20260810 23:10:28.327683 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:28.328009 14636 external_mini_cluster.cc:1428] Setting key c2695f3267ce4c87ba2ced0acf732471
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.2@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:28.334615 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.321907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:38957 (local address 127.14.75.62:40331)
0810 23:10:28.322071 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.322076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.322870 (+ 794us) server_negotiation.cc:407] Connection header received
0810 23:10:28.324171 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.324173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.324219 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.324295 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:28.325479 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.326002 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.327300 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.327660 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.327807 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.330691 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:28.330698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:28.330704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:28.330736 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:28.332290 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.332916 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.332919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.332920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.332993 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.333389 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.333391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.333393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.333616 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:28.333695 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.334389 (+ 694us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.334497 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:10:28.335897 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.322276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40331 (local address 127.14.75.2:38957)
0810 23:10:28.322705 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.322757 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.323950 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.324456 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.324462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.324830 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:28.325296 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.325309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.326115 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.326117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.327151 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.327170 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.327985 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.327989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.328205 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.328910 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:28.328926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:28.330526 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:28.332412 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.332420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.332436 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.332812 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.333081 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.333086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.333091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.333295 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.333730 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:28.333739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:28.334095 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.334338 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.334715 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260810 23:10:28.337069 2484 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40331
I20260810 23:10:28.337325 2484 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:28.337857 2484 heartbeater.cc:511] Master 127.14.75.62:40331 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.75.3@KRBTEST.COM; defaulting to no policy
I20260810 23:10:28.338879 2147 ts_manager.cc:194] Registered new tserver with Master: 5eac73b216eb43c6887766f426117a45 (127.14.75.2:36553)
I20260810 23:10:28.339427 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.2@KRBTEST.COM'} at 127.14.75.2:38957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
Entry for principal kudu/127.14.75.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab.
I20260810 23:10:28.356608 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:28.467242 2490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:28.467537 2490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:28.467614 2490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:28.471876 2490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:28.471961 2490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:28.472049 2490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:28.477380 2490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.3.keytab
--principal=kudu/127.14.75.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:40331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:28.478492 2490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:28.479519 2490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:28.488897 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.488976 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.488888 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:28.489105 2490 server_base.cc:1034] running on GCE node
I20260810 23:10:28.489656 2490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:28.490311 2490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:28.491504 2490 hybrid_clock.cc:648] HybridClock initialized: now 1786403428491493 us; error 31 us; skew 500 ppm
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:28.494338 2490 init.cc:377] Logged in from keytab as kudu/127.14.75.3@KRBTEST.COM (short username kudu)
I20260810 23:10:28.495206 2490 webserver.cc:460] Webserver started at http://127.14.75.3:39763/ using document root <none> and password file <none>
I20260810 23:10:28.495776 2490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:28.495891 2490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:28.496070 2490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:28.498104 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "8c2dbe0385244ad79675e3805d816c20"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "68c95cc2d04953b73901dccf9661bb6a"
server_key_iv: "ca08cd1e12a471a7b3094e576f44c42a"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.498603 2490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "8c2dbe0385244ad79675e3805d816c20"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "68c95cc2d04953b73901dccf9661bb6a"
server_key_iv: "ca08cd1e12a471a7b3094e576f44c42a"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.502167 2490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:10:28.504384 2505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.505628 2490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:28.505769 2490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "8c2dbe0385244ad79675e3805d816c20"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "68c95cc2d04953b73901dccf9661bb6a"
server_key_iv: "ca08cd1e12a471a7b3094e576f44c42a"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.505862 2490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:28.520247 2490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:28.523594 2490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:28.523836 2490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:28.524466 2490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:28.525322 2490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:28.525369 2490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.525460 2490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:28.525501 2490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.534507 2490 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:41415
I20260810 23:10:28.534543 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:41415 every 8 connection(s)
I20260810 23:10:28.535384 2490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:28.542976 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2490
I20260810 23:10:28.543138 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:28.543489 14636 external_mini_cluster.cc:1428] Setting key 42e376e8fa63799d132bf6e5bc4b9140
Aug 10 23:10:28 dist-test-slave-srgv krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403428, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.3@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:28.548770 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.537160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:53981 (local address 127.14.75.62:40331)
0810 23:10:28.537315 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.537318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.538164 (+ 846us) server_negotiation.cc:407] Connection header received
0810 23:10:28.539491 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.539495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.539550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.539635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:28.540758 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.541304 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.542502 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.542751 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.542862 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.545662 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:28.545665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:28.545667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:28.545689 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:28.547063 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.547516 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.547519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.547520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.547592 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.547884 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.547886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.547888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.548051 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:28.548124 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.548548 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.548647 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:10:28.549592 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.537559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:40331 (local address 127.14.75.3:53981)
0810 23:10:28.538017 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.538060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.539280 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.539811 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.539818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.540156 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:28.540609 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.540622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.541459 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.541467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.542371 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.542381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.542901 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.542905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.543069 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.543757 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:28.543773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:28.545497 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:28.547166 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.547170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.547180 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.547428 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.547668 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:28.547670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:28.547672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:28.547804 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:28.548135 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:28.548139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:28.548351 (+ 212us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.548490 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.548735 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260810 23:10:28.550614 2623 heartbeater.cc:347] Connected to a master server at 127.14.75.62:40331
I20260810 23:10:28.550863 2623 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:28.551327 2623 heartbeater.cc:511] Master 127.14.75.62:40331 requested a full tablet report, sending...
I20260810 23:10:28.552310 2147 ts_manager.cc:194] Registered new tserver with Master: 8c2dbe0385244ad79675e3805d816c20 (127.14.75.3:41415)
I20260810 23:10:28.552875 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.75.3@KRBTEST.COM'} at 127.14.75.3:53981
I20260810 23:10:28.557472 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:28.566488 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.559450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.14.75.62:40331)
0810 23:10:28.559601 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.559605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.559761 (+ 156us) server_negotiation.cc:407] Connection header received
0810 23:10:28.559851 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.559853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.559936 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.560015 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:28.560686 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.561190 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.562067 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.562294 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.562373 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.563718 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:28.563721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:28.563723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:28.563739 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:28.564994 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.565369 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.565371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.565373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.565519 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:28.565769 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:28.565771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:28.565772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:28.565977 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:28.566077 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.566270 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.566369 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:10:28.569694 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:28.572911 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:28.587641 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.14.75.3:41415)
0810 23:10:28.582437 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.582449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.582472 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:10:28.582575 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.582582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.582801 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.582964 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:28.583507 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584339 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.586346 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.586894 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.587028 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.587181 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.587359 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.587466 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":159,"threads_started":1}
I20260810 23:10:28.587667 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36553 (local address 127.0.0.1:54822)
0810 23:10:28.581736 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.581792 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.581973 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.582834 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.582840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.582866 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:28.583004 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.583011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584713 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.584716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.585993 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.586003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.587106 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.587110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.587212 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.587248 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.587307 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.587467 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":104,"spinlock_wait_cycles":2560,"thread_start_us":124,"threads_started":1}
I20260810 23:10:28.587859 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42137 (local address 127.0.0.1:55456)
0810 23:10:28.581736 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.581793 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.581937 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.583282 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.583284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.583301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:28.583428 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.583436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584799 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.584801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.586196 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.586207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.587430 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.587434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.587524 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.587549 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.587631 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.587704 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":151,"spinlock_wait_cycles":6784,"thread_start_us":70,"threads_started":1}
I20260810 23:10:28.587641 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:41415 (local address 127.0.0.1:37632)
0810 23:10:28.582317 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.582331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.582409 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.583163 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.583167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.583188 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:28.583365 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.583373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584472 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.584478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.586196 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.586207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.587031 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.587034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.587151 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.587188 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.587329 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.587474 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":215,"thread_start_us":111,"threads_started":1}
I20260810 23:10:28.588296 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.14.75.1:42137)
0810 23:10:28.582848 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.582864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.582884 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:10:28.582952 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.582956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.583159 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.583298 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:28.583545 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584647 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.586327 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.587125 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.587216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.587366 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.588039 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.588146 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":153,"threads_started":1}
I20260810 23:10:28.589368 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.581805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.14.75.2:36553)
0810 23:10:28.582274 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.582288 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.582312 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:10:28.582395 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.582400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.582652 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.583173 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:28.583351 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.584573 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.586119 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.586951 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.588145 (+ 1194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.589062 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.589112 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.589204 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":124,"threads_started":1}
I20260810 23:10:28.590821 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 7852a243c08442efb386c9092295be3f (DEFAULT_TABLE table=test-table [id=ea6e7f24b2b44d5f905f551bc9726068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:28.590981 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 7852a243c08442efb386c9092295be3f (DEFAULT_TABLE table=test-table [id=ea6e7f24b2b44d5f905f551bc9726068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:28.591830 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852a243c08442efb386c9092295be3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:28.592041 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852a243c08442efb386c9092295be3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:28.591769 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 7852a243c08442efb386c9092295be3f (DEFAULT_TABLE table=test-table [id=ea6e7f24b2b44d5f905f551bc9726068]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:28.592697 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852a243c08442efb386c9092295be3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:28.598219 2640 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Bootstrap starting.
I20260810 23:10:28.599646 2641 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Bootstrap starting.
I20260810 23:10:28.600626 2640 tablet_bootstrap.cc:654] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:28.601786 2642 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: Bootstrap starting.
I20260810 23:10:28.601827 2641 tablet_bootstrap.cc:654] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:28.602533 2640 log.cc:824] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:28.603053 2641 log.cc:824] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:28.603732 2642 tablet_bootstrap.cc:654] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:28.604496 2640 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: No bootstrap required, opened a new log
I20260810 23:10:28.604686 2640 ts_tablet_manager.cc:1397] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 23:10:28.604828 2642 log.cc:824] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:28.606666 2642 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: No bootstrap required, opened a new log
I20260810 23:10:28.606878 2642 ts_tablet_manager.cc:1397] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:10:28.607105 2641 tablet_bootstrap.cc:492] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: No bootstrap required, opened a new log
I20260810 23:10:28.607288 2641 ts_tablet_manager.cc:1397] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:10:28.608841 2640 raft_consensus.cc:348] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.609126 2640 raft_consensus.cc:374] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:28.609187 2640 raft_consensus.cc:729] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2dbe0385244ad79675e3805d816c20, State: Initialized, Role: FOLLOWER
I20260810 23:10:28.609741 2640 consensus_queue.cc:260] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.610796 2640 ts_tablet_manager.cc:1428] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:10:28.610872 2642 raft_consensus.cc:348] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.611142 2642 raft_consensus.cc:374] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:28.611202 2623 heartbeater.cc:503] Master 127.14.75.62:40331 was elected leader, sending a full tablet report...
I20260810 23:10:28.611207 2642 raft_consensus.cc:729] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003d5e3f29d0449fa1b6bba8f586a721, State: Initialized, Role: FOLLOWER
I20260810 23:10:28.611080 2641 raft_consensus.cc:348] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.611348 2641 raft_consensus.cc:374] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:28.611409 2641 raft_consensus.cc:729] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eac73b216eb43c6887766f426117a45, State: Initialized, Role: FOLLOWER
I20260810 23:10:28.611734 2642 consensus_queue.cc:260] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.611770 2641 consensus_queue.cc:260] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.612363 2344 heartbeater.cc:503] Master 127.14.75.62:40331 was elected leader, sending a full tablet report...
I20260810 23:10:28.612627 2642 ts_tablet_manager.cc:1428] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:10:28.612689 2641 ts_tablet_manager.cc:1428] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:10:28.612987 2484 heartbeater.cc:503] Master 127.14.75.62:40331 was elected leader, sending a full tablet report...
I20260810 23:10:28.716188 2647 raft_consensus.cc:482] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:28.716470 2647 raft_consensus.cc:504] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.717803 2647 leader_election.cc:288] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eac73b216eb43c6887766f426117a45 (127.14.75.2:36553), 8c2dbe0385244ad79675e3805d816c20 (127.14.75.3:41415)
I20260810 23:10:28.722079 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.718176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:50265 (local address 127.14.75.3:41415)
0810 23:10:28.718318 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.718325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.718594 (+ 269us) server_negotiation.cc:407] Connection header received
0810 23:10:28.718668 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.718671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.718740 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.718826 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:28.719189 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.719772 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.720967 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.721391 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.721557 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.721631 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.721815 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.721888 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:10:28.722062 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.718044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:36553 (local address 127.14.75.1:55139)
0810 23:10:28.718675 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.718689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.718798 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.719067 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.719070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.719084 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:28.719242 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.719249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.719981 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.719983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.720839 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.720848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.721599 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.721603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.721715 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.721733 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.721829 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.721895 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":39,"thread_start_us":172,"threads_started":1}
I20260810 23:10:28.722074 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.718028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:41415 (local address 127.14.75.1:50265)
0810 23:10:28.718479 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.718491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:28.718592 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:28.718857 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:28.718860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:28.718878 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:28.719061 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.719069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.719895 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.719901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.720821 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:28.720830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.721575 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.721579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:28.721667 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.721697 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:10:28.721768 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:10:28.721857 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":49,"thread_start_us":202,"threads_started":1}
I20260810 23:10:28.722808 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.718294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:55139 (local address 127.14.75.2:36553)
0810 23:10:28.718453 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.718457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.718812 (+ 355us) server_negotiation.cc:407] Connection header received
0810 23:10:28.718875 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.718879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.718927 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.719050 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:28.719362 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.719864 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.720997 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.721456 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.721584 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.721642 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.721863 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.722657 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:10:28.723100 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852a243c08442efb386c9092295be3f" candidate_uuid: "003d5e3f29d0449fa1b6bba8f586a721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2dbe0385244ad79675e3805d816c20" is_pre_election: true
I20260810 23:10:28.723363 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852a243c08442efb386c9092295be3f" candidate_uuid: "003d5e3f29d0449fa1b6bba8f586a721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eac73b216eb43c6887766f426117a45" is_pre_election: true
I20260810 23:10:28.723448 2575 raft_consensus.cc:2454] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 003d5e3f29d0449fa1b6bba8f586a721 in term 0.
I20260810 23:10:28.723665 2437 raft_consensus.cc:2454] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 003d5e3f29d0449fa1b6bba8f586a721 in term 0.
I20260810 23:10:28.723912 2227 leader_election.cc:302] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003d5e3f29d0449fa1b6bba8f586a721, 8c2dbe0385244ad79675e3805d816c20; no voters:
I20260810 23:10:28.724261 2647 raft_consensus.cc:2781] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:28.724349 2647 raft_consensus.cc:482] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:28.724392 2647 raft_consensus.cc:3037] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:28.725525 2647 raft_consensus.cc:504] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.725852 2647 leader_election.cc:288] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [CANDIDATE]: Term 1 election: Requested vote from peers 5eac73b216eb43c6887766f426117a45 (127.14.75.2:36553), 8c2dbe0385244ad79675e3805d816c20 (127.14.75.3:41415)
I20260810 23:10:28.726239 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852a243c08442efb386c9092295be3f" candidate_uuid: "003d5e3f29d0449fa1b6bba8f586a721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eac73b216eb43c6887766f426117a45"
I20260810 23:10:28.726224 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852a243c08442efb386c9092295be3f" candidate_uuid: "003d5e3f29d0449fa1b6bba8f586a721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2dbe0385244ad79675e3805d816c20"
I20260810 23:10:28.726359 2437 raft_consensus.cc:3037] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:28.726359 2575 raft_consensus.cc:3037] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:28.727746 2575 raft_consensus.cc:2454] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003d5e3f29d0449fa1b6bba8f586a721 in term 1.
I20260810 23:10:28.727746 2437 raft_consensus.cc:2454] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003d5e3f29d0449fa1b6bba8f586a721 in term 1.
I20260810 23:10:28.728049 2227 leader_election.cc:302] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003d5e3f29d0449fa1b6bba8f586a721, 8c2dbe0385244ad79675e3805d816c20; no voters:
I20260810 23:10:28.728223 2647 raft_consensus.cc:2781] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:28.728451 2647 raft_consensus.cc:686] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [term 1 LEADER]: Becoming Leader. State: Replica: 003d5e3f29d0449fa1b6bba8f586a721, State: Running, Role: LEADER
I20260810 23:10:28.728801 2647 consensus_queue.cc:237] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } }
I20260810 23:10:28.731988 2145 catalog_manager.cc:5495] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 reported cstate change: term changed from 0 to 1, leader changed from <none> to 003d5e3f29d0449fa1b6bba8f586a721 (127.14.75.1). New cstate: current_term: 1 leader_uuid: "003d5e3f29d0449fa1b6bba8f586a721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "003d5e3f29d0449fa1b6bba8f586a721" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 42137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:28.743243 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.740059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.14.75.1:42137)
0810 23:10:28.740181 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:28.740186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:28.740199 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:10:28.740323 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:28.740325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:28.740377 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:28.740461 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:28.740678 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.741163 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.742020 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:28.742228 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:28.742304 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:28.742552 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:28.742952 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:28.743041 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:28.743095 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:10:28.743140 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:28.748744 2575 raft_consensus.cc:1261] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 1 FOLLOWER]: Refusing update from remote peer 003d5e3f29d0449fa1b6bba8f586a721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:28.748749 2437 raft_consensus.cc:1261] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 1 FOLLOWER]: Refusing update from remote peer 003d5e3f29d0449fa1b6bba8f586a721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:28.749490 2647 consensus_queue.cc:1035] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eac73b216eb43c6887766f426117a45" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 36553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:28.749691 2651 consensus_queue.cc:1035] T 7852a243c08442efb386c9092295be3f P 003d5e3f29d0449fa1b6bba8f586a721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2dbe0385244ad79675e3805d816c20" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:28.758476 2656 mvcc.cc:205] Tried to move back new op lower bound from 7317108444148228096 to 7317108444076302336. Current Snapshot: MvccSnapshot[applied={T|T < 7317108444148228096}]
I20260810 23:10:28.759063 2655 mvcc.cc:205] Tried to move back new op lower bound from 7317108444148228096 to 7317108444076302336. Current Snapshot: MvccSnapshot[applied={T|T < 7317108444148228096}]
I20260810 23:10:28.760982 2654 mvcc.cc:205] Tried to move back new op lower bound from 7317108444148228096 to 7317108444076302336. Current Snapshot: MvccSnapshot[applied={T|T < 7317108444148228096 or (T in {7317108444148228096})}]
I20260810 23:10:28.765587 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:28.765751 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:28.768079 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b601b46e073b4433b094ac0d1a8d6f95: Sending DeleteTablet for 3 replicas of tablet 7852a243c08442efb386c9092295be3f
I20260810 23:10:28.768857 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 7852a243c08442efb386c9092295be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:28 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:54822
I20260810 23:10:28.768915 2555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7852a243c08442efb386c9092295be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:28 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:37632
I20260810 23:10:28.769357 2664 tablet_replica.cc:330] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: stopping tablet replica
I20260810 23:10:28.769512 2663 tablet_replica.cc:330] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: stopping tablet replica
I20260810 23:10:28.769693 2664 raft_consensus.cc:2229] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:28.769788 2663 raft_consensus.cc:2229] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:28.769711 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 7852a243c08442efb386c9092295be3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:28 UTC) from {username='kudu', principal='kudu/127.14.75.62@KRBTEST.COM'} at 127.0.0.1:55456
I20260810 23:10:28.769999 2663 raft_consensus.cc:2258] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:28.769999 2664 raft_consensus.cc:2258] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:28.770318 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2209
I20260810 23:10:28.771575 2664 ts_tablet_manager.cc:1905] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:28.771700 2663 ts_tablet_manager.cc:1905] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:28.774305 2663 ts_tablet_manager.cc:1918] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:28.774305 2664 ts_tablet_manager.cc:1918] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:28.774413 2663 log.cc:1196] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/7852a243c08442efb386c9092295be3f
I20260810 23:10:28.774413 2664 log.cc:1196] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/7852a243c08442efb386c9092295be3f
I20260810 23:10:28.774775 2664 ts_tablet_manager.cc:1939] T 7852a243c08442efb386c9092295be3f P 8c2dbe0385244ad79675e3805d816c20: Deleting consensus metadata
I20260810 23:10:28.774775 2663 ts_tablet_manager.cc:1939] T 7852a243c08442efb386c9092295be3f P 5eac73b216eb43c6887766f426117a45: Deleting consensus metadata
I20260810 23:10:28.776041 2131 catalog_manager.cc:4841] TS 5eac73b216eb43c6887766f426117a45 (127.14.75.2:36553): tablet 7852a243c08442efb386c9092295be3f (table test-table [id=ea6e7f24b2b44d5f905f551bc9726068]) successfully deleted
I20260810 23:10:28.776096 2133 catalog_manager.cc:4841] TS 8c2dbe0385244ad79675e3805d816c20 (127.14.75.3:41415): tablet 7852a243c08442efb386c9092295be3f (table test-table [id=ea6e7f24b2b44d5f905f551bc9726068]) successfully deleted
W20260810 23:10:28.778539 2132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:42137 (error 108)
I20260810 23:10:28.779346 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2349
I20260810 23:10:28.779610 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:28.779289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:42137 (local address 127.0.0.1:55478)
0810 23:10:28.779417 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:28.779514 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:10:28.781364 2132 catalog_manager.cc:4584] TS 003d5e3f29d0449fa1b6bba8f586a721 (127.14.75.1:42137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7852a243c08442efb386c9092295be3f: Network error: Client connection negotiation failed: client connection to 127.14.75.1:42137: connect: Connection refused (error 111)
I20260810 23:10:28.785753 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2490
I20260810 23:10:28.792191 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2115
2026-08-10T23:10:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2360 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:28Z Disabled control of system clock
I20260810 23:10:28.824134 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45513
--webserver_interface=127.14.75.62
--webserver_port=0
--builtin_ntp_servers=127.14.75.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:28.934111 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:28.934355 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:28.934391 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:28.938236 2670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:28.938282 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:28.938297 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:28.938313 2670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:28.938328 2670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:28.942934 2670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:45513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:28.943965 2670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:28.945257 2670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:28.953446 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:28.953454 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:28.953586 2670 server_base.cc:1034] running on GCE node
W20260810 23:10:28.953893 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:28.954452 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:28.955466 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:28.956629 2670 hybrid_clock.cc:648] HybridClock initialized: now 1786403428956609 us; error 33 us; skew 500 ppm
I20260810 23:10:28.958601 2670 webserver.cc:460] Webserver started at http://127.14.75.62:33209/ using document root <none> and password file <none>
I20260810 23:10:28.959172 2670 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:28.959301 2670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:28.959553 2670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:28.961949 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "f496eb88edb34673a7af9aad04a7d04d"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "c1779b717ee1ec490edbf1defebe3508"
server_key_iv: "8c4583252b598a69780370f28f670011"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.962461 2670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "f496eb88edb34673a7af9aad04a7d04d"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "c1779b717ee1ec490edbf1defebe3508"
server_key_iv: "8c4583252b598a69780370f28f670011"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.965958 2670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:10:28.968202 2684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:28.969388 2670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:28.969561 2670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "f496eb88edb34673a7af9aad04a7d04d"
format_stamp: "Formatted at 2026-08-10 23:10:28 on dist-test-slave-srgv"
server_key: "c1779b717ee1ec490edbf1defebe3508"
server_key_iv: "8c4583252b598a69780370f28f670011"
server_key_version: "encryptionkey@0"
I20260810 23:10:28.969666 2670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:28.999027 2670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:28.999785 2670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:29.000011 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:29.007203 2670 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:45513
I20260810 23:10:29.007215 2738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:45513 every 8 connection(s)
I20260810 23:10:29.008181 2670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:29.010210 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2670
I20260810 23:10:29.010342 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:29.010633 14636 external_mini_cluster.cc:1428] Setting key eb5db15b54cbc66324f1dbf4d4941f22
I20260810 23:10:29.011713 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:29.018777 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: Bootstrap starting.
I20260810 23:10:29.020998 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.012087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.14.75.62:45513)
0810 23:10:29.012761 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.012783 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.012826 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:10:29.013759 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.013783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.014270 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.014643 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:29.015103 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.016524 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.017546 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.018229 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.018389 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.019306 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:29.019311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:29.019323 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:29.019695 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:29.019750 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:29.019772 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:29.019847 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.020022 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.020320 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260810 23:10:29.021179 2739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:29.022075 2739 log.cc:824] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:29.024139 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: No bootstrap required, opened a new log
I20260810 23:10:29.027457 2739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } }
I20260810 23:10:29.027685 2739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:29.027765 2739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f496eb88edb34673a7af9aad04a7d04d, State: Initialized, Role: FOLLOWER
I20260810 23:10:29.028205 2739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } }
I20260810 23:10:29.028348 2739 raft_consensus.cc:388] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:29.028434 2739 raft_consensus.cc:482] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:29.028532 2739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:29.029630 2739 raft_consensus.cc:504] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } }
I20260810 23:10:29.029946 2739 leader_election.cc:302] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f496eb88edb34673a7af9aad04a7d04d; no voters:
I20260810 23:10:29.030259 2739 leader_election.cc:288] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:29.030417 2744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:29.030671 2744 raft_consensus.cc:686] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [term 1 LEADER]: Becoming Leader. State: Replica: f496eb88edb34673a7af9aad04a7d04d, State: Running, Role: LEADER
I20260810 23:10:29.031010 2744 consensus_queue.cc:237] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } }
I20260810 23:10:29.031491 2739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:29.032796 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f496eb88edb34673a7af9aad04a7d04d. Latest consensus state: current_term: 1 leader_uuid: "f496eb88edb34673a7af9aad04a7d04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } } }
I20260810 23:10:29.032923 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:29.033192 2745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f496eb88edb34673a7af9aad04a7d04d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f496eb88edb34673a7af9aad04a7d04d" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45513 } } }
I20260810 23:10:29.033293 2745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:29.034083 2755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:29.036718 2755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:29.042222 2755 catalog_manager.cc:1347] Generated new cluster ID: be10a7ad2d8343b6b69d3c60f1ada182
I20260810 23:10:29.042289 2755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:29.090570 2755 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:29.091842 2755 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:29.098990 2755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: Generated new TSK 0
I20260810 23:10:29.099753 2755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:29.109820 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.1:0
--local_ip_for_outbound_sockets=127.14.75.1
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--builtin_ntp_servers=127.14.75.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:10:29.222744 2766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:29.223016 2766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:29.223060 2766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:29.227183 2766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:29.227241 2766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:29.227301 2766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.1
I20260810 23:10:29.232167 2766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.75.1
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.1
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:29.233318 2766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:29.234742 2766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:29.244607 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:29.244607 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:29.244643 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:29.244855 2766 server_base.cc:1034] running on GCE node
I20260810 23:10:29.245493 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:29.246197 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:29.247396 2766 hybrid_clock.cc:648] HybridClock initialized: now 1786403429247362 us; error 45 us; skew 500 ppm
I20260810 23:10:29.249330 2766 webserver.cc:460] Webserver started at http://127.14.75.1:42117/ using document root <none> and password file <none>
I20260810 23:10:29.249950 2766 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:29.250044 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:29.250288 2766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:29.252339 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data/instance:
uuid: "679aa5f3c18149b29c9f80130e9bb36e"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "ddfded7e4e09e12e9740e11f43ef399d"
server_key_iv: "76986bb686b96035c7e79254a6f943d4"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.252895 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance:
uuid: "679aa5f3c18149b29c9f80130e9bb36e"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "ddfded7e4e09e12e9740e11f43ef399d"
server_key_iv: "76986bb686b96035c7e79254a6f943d4"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.256564 2766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:10:29.258993 2782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.260290 2766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:29.260438 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal
uuid: "679aa5f3c18149b29c9f80130e9bb36e"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "ddfded7e4e09e12e9740e11f43ef399d"
server_key_iv: "76986bb686b96035c7e79254a6f943d4"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.260555 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:29.270563 2766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:29.271226 2766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:29.271420 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:29.272003 2766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:29.272979 2766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:29.273049 2766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.273119 2766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:29.273166 2766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.282166 2766 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.1:36821
I20260810 23:10:29.282191 2900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.1:36821 every 8 connection(s)
I20260810 23:10:29.283099 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/data/info.pb
I20260810 23:10:29.286619 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2766
I20260810 23:10:29.286724 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-0/wal/instance
I20260810 23:10:29.286998 14636 external_mini_cluster.cc:1428] Setting key f7d7c7546423cb04bd6acb3569c513b7
I20260810 23:10:29.288942 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.2:0
--local_ip_for_outbound_sockets=127.14.75.2
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--builtin_ntp_servers=127.14.75.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:29.292785 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.284939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:57527 (local address 127.14.75.62:45513)
0810 23:10:29.285080 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.285084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.286183 (+ 1099us) server_negotiation.cc:407] Connection header received
0810 23:10:29.287786 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.287790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.287848 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.287953 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:29.288939 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.289630 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.290770 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.291002 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.291120 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.292259 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:29.292271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:29.292280 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:29.292294 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:29.292315 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:29.292323 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:29.292407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.292570 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.292640 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:10:29.293706 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.285313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45513 (local address 127.14.75.1:57527)
0810 23:10:29.286026 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.286066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.287563 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.288125 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.288132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.288210 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:29.288792 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.288806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.289774 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.289782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.290638 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.290647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.291164 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.291168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.291335 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.291990 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:29.291991 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:29.292033 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:29.292098 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:29.292408 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:29.292420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:29.292430 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.292569 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.292863 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":134,"threads_started":1}
I20260810 23:10:29.295001 2901 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45513
I20260810 23:10:29.295258 2901 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:29.295799 2901 heartbeater.cc:511] Master 127.14.75.62:45513 requested a full tablet report, sending...
I20260810 23:10:29.297549 2700 ts_manager.cc:194] Registered new tserver with Master: 679aa5f3c18149b29c9f80130e9bb36e (127.14.75.1:36821)
I20260810 23:10:29.299302 2700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.1:57527
W20260810 23:10:29.401983 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:29.402259 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:29.402302 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:29.406405 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:29.406463 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:29.406520 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.2
I20260810 23:10:29.411350 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.75.2
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.2
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:29.412447 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:29.413892 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:29.423554 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:29.423547 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:29.423547 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:29.423825 2905 server_base.cc:1034] running on GCE node
I20260810 23:10:29.424337 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:29.425043 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:29.426241 2905 hybrid_clock.cc:648] HybridClock initialized: now 1786403429426228 us; error 36 us; skew 500 ppm
I20260810 23:10:29.433871 2905 webserver.cc:460] Webserver started at http://127.14.75.2:38607/ using document root <none> and password file <none>
I20260810 23:10:29.434437 2905 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:29.434502 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:29.434674 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:29.436702 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data/instance:
uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "021c3fe7fe7bab18435fdd985093030e"
server_key_iv: "b470209cfa18c69ece830ae8b98a7871"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.437207 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance:
uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "021c3fe7fe7bab18435fdd985093030e"
server_key_iv: "b470209cfa18c69ece830ae8b98a7871"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.440987 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:10:29.443403 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.444700 2905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:29.444795 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal
uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "021c3fe7fe7bab18435fdd985093030e"
server_key_iv: "b470209cfa18c69ece830ae8b98a7871"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.444881 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:29.464964 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:29.465806 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:29.465983 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:29.466533 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:29.467427 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:29.467474 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.467514 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:29.467535 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.477010 2905 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.2:45399
I20260810 23:10:29.477031 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.2:45399 every 8 connection(s)
I20260810 23:10:29.478003 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/data/info.pb
I20260810 23:10:29.486167 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 2905
I20260810 23:10:29.486310 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal/instance
I20260810 23:10:29.486699 14636 external_mini_cluster.cc:1428] Setting key 283615cdd45181326975f7b27ab92924
I20260810 23:10:29.486930 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.479742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.2:60265 (local address 127.14.75.62:45513)
0810 23:10:29.479883 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.479887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.480842 (+ 955us) server_negotiation.cc:407] Connection header received
0810 23:10:29.482295 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.482298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.482347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.482414 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:29.483275 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.483779 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.484882 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.485082 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.485159 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.486385 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:29.486389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:29.486391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:29.486406 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:29.486430 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:29.486441 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:29.486524 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.486695 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.486778 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:10:29.487941 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.480068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45513 (local address 127.14.75.2:60265)
0810 23:10:29.480712 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.480753 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.482081 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.482585 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.482592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.482672 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:29.483149 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.483160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.483890 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.483893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.484778 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.484787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.485223 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.485226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.485368 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.486076 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:29.486078 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:29.486120 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:29.486183 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:29.486546 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:29.486554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:29.486559 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.486664 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.486946 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":122,"threads_started":1}
I20260810 23:10:29.488927 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.75.3:0
--local_ip_for_outbound_sockets=127.14.75.3
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--builtin_ntp_servers=127.14.75.20:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:10:29.489220 3037 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45513
I20260810 23:10:29.489500 3037 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:29.490021 3037 heartbeater.cc:511] Master 127.14.75.62:45513 requested a full tablet report, sending...
I20260810 23:10:29.491087 2700 ts_manager.cc:194] Registered new tserver with Master: 8b32fb1a3fbd4d3ea6ee575be1f2a11c (127.14.75.2:45399)
I20260810 23:10:29.491740 2700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.2:60265
W20260810 23:10:29.606135 3040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:29.606408 3040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:29.606447 3040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:29.610467 3040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:29.610523 3040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:29.610582 3040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.75.3
I20260810 23:10:29.615285 3040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.75.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.75.3
--webserver_port=0
--tserver_master_addrs=127.14.75.62:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.75.3
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:29.616305 3040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:29.617741 3040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:29.627460 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:29.627779 3040 server_base.cc:1034] running on GCE node
W20260810 23:10:29.627460 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:29.627472 3049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:29.628300 3040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:29.628957 3040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:29.630187 3040 hybrid_clock.cc:648] HybridClock initialized: now 1786403429630170 us; error 26 us; skew 500 ppm
I20260810 23:10:29.632503 3040 webserver.cc:460] Webserver started at http://127.14.75.3:45235/ using document root <none> and password file <none>
I20260810 23:10:29.633157 3040 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:29.633257 3040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:29.633584 3040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:29.635838 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data/instance:
uuid: "96f6aa4fe5154919bed78e17d3f42bb4"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "f7a7f9f2e50d5f65db637085de8f1e25"
server_key_iv: "f508f32f5d579eb1ff36017f9777e676"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.636441 3040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance:
uuid: "96f6aa4fe5154919bed78e17d3f42bb4"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "f7a7f9f2e50d5f65db637085de8f1e25"
server_key_iv: "f508f32f5d579eb1ff36017f9777e676"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.640298 3040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:10:29.642828 3055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.644063 3040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:29.644207 3040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal
uuid: "96f6aa4fe5154919bed78e17d3f42bb4"
format_stamp: "Formatted at 2026-08-10 23:10:29 on dist-test-slave-srgv"
server_key: "f7a7f9f2e50d5f65db637085de8f1e25"
server_key_iv: "f508f32f5d579eb1ff36017f9777e676"
server_key_version: "encryptionkey@0"
I20260810 23:10:29.644337 3040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:29.655905 3040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:29.656623 3040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:29.656833 3040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:29.657493 3040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:10:29.658452 3040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:10:29.658525 3040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.658598 3040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:10:29.658641 3040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:29.667695 3040 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.3:36721
I20260810 23:10:29.667724 3172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.3:36721 every 8 connection(s)
I20260810 23:10:29.668617 3040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/data/info.pb
I20260810 23:10:29.676290 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3040
I20260810 23:10:29.676430 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal/instance
I20260810 23:10:29.676699 14636 external_mini_cluster.cc:1428] Setting key dd8dd3d8cf27754ff1495aaff4a5340f
I20260810 23:10:29.678159 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.670499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:57633 (local address 127.14.75.62:45513)
0810 23:10:29.670648 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.670651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.671703 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 23:10:29.673186 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.673190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.673251 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.673365 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:29.674305 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.674838 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.676057 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.676280 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.676395 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.677641 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:29.677644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:29.677646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:29.677658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:29.677678 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:29.677686 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:29.677785 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.677957 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.678022 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:10:29.679114 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.670896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:45513 (local address 127.14.75.3:57633)
0810 23:10:29.671549 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.671590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.672974 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.673567 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.673573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.673657 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:29.674145 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.674157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.674973 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.674981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.675911 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.675920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.676442 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.676446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.676618 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.677320 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:10:29.677322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:29.677362 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:10:29.677465 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:29.677792 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:29.677804 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:29.677810 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.677954 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.678244 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260810 23:10:29.680186 3173 heartbeater.cc:347] Connected to a master server at 127.14.75.62:45513
I20260810 23:10:29.680397 3173 heartbeater.cc:464] Registering TS with master...
I20260810 23:10:29.680864 3173 heartbeater.cc:511] Master 127.14.75.62:45513 requested a full tablet report, sending...
I20260810 23:10:29.681864 2700 ts_manager.cc:194] Registered new tserver with Master: 96f6aa4fe5154919bed78e17d3f42bb4 (127.14.75.3:36721)
I20260810 23:10:29.682488 2700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.75.3:57633
I20260810 23:10:29.691076 14636 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:10:29.696908 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.693197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.14.75.62:45513)
0810 23:10:29.693354 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.693358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.693580 (+ 222us) server_negotiation.cc:407] Connection header received
0810 23:10:29.693689 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.693691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.693733 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.693793 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:29.694074 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.694573 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.695385 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.695584 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.695656 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.696486 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:29.696488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:29.696491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:29.696501 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:29.696515 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:29.696522 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:29.696575 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.696676 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.696719 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:10:29.699910 2700 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:10:29.702975 2700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:10:29.717259 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.711304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.14.75.3:36721)
0810 23:10:29.711849 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.711863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.711891 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:10:29.711968 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.711973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.712183 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.712348 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.712654 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.713792 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.715406 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.716182 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.716273 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.716478 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.716952 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.717066 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":154,"threads_started":1}
I20260810 23:10:29.717245 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.711743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.14.75.2:45399)
0810 23:10:29.712509 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.712522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.712549 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:10:29.712625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.712629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.712792 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.712950 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.713191 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.714242 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.715802 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.716467 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.716596 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.716782 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.717006 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.717089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":114,"threads_started":1}
I20260810 23:10:29.717217 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.710780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36721 (local address 127.0.0.1:59140)
0810 23:10:29.711122 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.711168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.711302 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.712335 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.712340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.712364 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.712507 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.712516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.713901 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.713903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.715244 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.715256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.716585 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.716588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.716726 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.716770 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.716893 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.717055 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260810 23:10:29.717643 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.710872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:45399 (local address 127.0.0.1:47046)
0810 23:10:29.712124 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.712137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.712244 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.712931 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.712934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.712951 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.713056 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.713066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.714375 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.714379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.715669 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.715679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.716598 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.716602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.716787 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.716823 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.716929 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.717519 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":505,"thread_start_us":110,"threads_started":1}
I20260810 23:10:29.718006 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.711701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.14.75.1:36821)
0810 23:10:29.712075 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.712087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.712825 (+ 738us) server_negotiation.cc:407] Connection header received
0810 23:10:29.712901 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.712906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.713070 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.713203 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.713617 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.714615 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.716164 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.716957 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.717054 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.717214 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.717717 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.717850 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260810 23:10:29.718566 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.711509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:36821 (local address 127.0.0.1:39470)
0810 23:10:29.712592 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.712606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.712712 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.713341 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.713344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.713362 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.713490 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.713496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.714941 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.714944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.716013 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.716024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.717281 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.717285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.717378 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.717432 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.717515 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.717597 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260810 23:10:29.720793 2968 tablet_service.cc:1467] Processing CreateTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 (DEFAULT_TABLE table=test-table [id=247dc3f7a7364baa9e3b2fcc83310a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:29.720789 3104 tablet_service.cc:1467] Processing CreateTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 (DEFAULT_TABLE table=test-table [id=247dc3f7a7364baa9e3b2fcc83310a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:29.721179 2831 tablet_service.cc:1467] Processing CreateTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 (DEFAULT_TABLE table=test-table [id=247dc3f7a7364baa9e3b2fcc83310a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:10:29.721987 2968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae7c5ac0a0d40a38a14c3bb810d2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:29.721987 3104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae7c5ac0a0d40a38a14c3bb810d2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:29.722159 2831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eae7c5ac0a0d40a38a14c3bb810d2166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:29.728443 3191 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Bootstrap starting.
I20260810 23:10:29.729913 3193 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Bootstrap starting.
I20260810 23:10:29.730822 3191 tablet_bootstrap.cc:654] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:29.731739 3191 log.cc:824] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:29.731920 3193 tablet_bootstrap.cc:654] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:29.732640 3193 log.cc:824] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:29.733314 3192 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Bootstrap starting.
I20260810 23:10:29.733865 3191 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: No bootstrap required, opened a new log
I20260810 23:10:29.734063 3191 ts_tablet_manager.cc:1397] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:10:29.735173 3192 tablet_bootstrap.cc:654] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:29.735836 3192 log.cc:824] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:29.737607 3193 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: No bootstrap required, opened a new log
I20260810 23:10:29.737814 3193 ts_tablet_manager.cc:1397] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:10:29.737805 3191 raft_consensus.cc:348] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.738085 3191 raft_consensus.cc:374] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:29.738138 3191 raft_consensus.cc:729] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679aa5f3c18149b29c9f80130e9bb36e, State: Initialized, Role: FOLLOWER
I20260810 23:10:29.738595 3191 consensus_queue.cc:260] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.738709 3192 tablet_bootstrap.cc:492] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: No bootstrap required, opened a new log
I20260810 23:10:29.738888 3192 ts_tablet_manager.cc:1397] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 23:10:29.739244 2901 heartbeater.cc:503] Master 127.14.75.62:45513 was elected leader, sending a full tablet report...
I20260810 23:10:29.739566 3191 ts_tablet_manager.cc:1428] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:10:29.742266 3193 raft_consensus.cc:348] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.742548 3193 raft_consensus.cc:374] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:29.742606 3193 raft_consensus.cc:729] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b32fb1a3fbd4d3ea6ee575be1f2a11c, State: Initialized, Role: FOLLOWER
I20260810 23:10:29.743059 3193 consensus_queue.cc:260] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.743114 3192 raft_consensus.cc:348] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.743377 3192 raft_consensus.cc:374] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:29.743443 3192 raft_consensus.cc:729] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f6aa4fe5154919bed78e17d3f42bb4, State: Initialized, Role: FOLLOWER
I20260810 23:10:29.743846 3192 consensus_queue.cc:260] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.744594 3173 heartbeater.cc:503] Master 127.14.75.62:45513 was elected leader, sending a full tablet report...
I20260810 23:10:29.744941 3192 ts_tablet_manager.cc:1428] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 23:10:29.744941 3193 ts_tablet_manager.cc:1428] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:10:29.745307 3037 heartbeater.cc:503] Master 127.14.75.62:45513 was elected leader, sending a full tablet report...
W20260810 23:10:29.784339 2902 tablet.cc:2369] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:10:29.912892 3199 raft_consensus.cc:482] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:29.913182 3199 raft_consensus.cc:504] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.914366 3199 leader_election.cc:288] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b32fb1a3fbd4d3ea6ee575be1f2a11c (127.14.75.2:45399), 679aa5f3c18149b29c9f80130e9bb36e (127.14.75.1:36821)
I20260810 23:10:29.917626 3197 raft_consensus.cc:482] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:10:29.917832 3197 raft_consensus.cc:504] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.918761 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.914626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:45399 (local address 127.14.75.3:39135)
0810 23:10:29.914761 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.914778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.914884 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.915296 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.915302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.915325 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.915461 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.915470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.916411 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.916414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.917307 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.917316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.918259 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.918263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.918377 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.918453 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.918533 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.918616 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":26}
I20260810 23:10:29.918780 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.914758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:39135 (local address 127.14.75.2:45399)
0810 23:10:29.914919 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.914925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.914945 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:10:29.915030 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.915033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.915087 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.915201 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.915617 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.916307 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.917487 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.918134 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.918254 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.918316 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.918578 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.918655 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:29.918872 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.915127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.3:40309 (local address 127.14.75.1:36821)
0810 23:10:29.915283 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.915287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.915455 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:10:29.915541 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.915547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.915607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.915665 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.915967 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.916587 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.917649 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.918186 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.918304 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.918360 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.918673 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.918746 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:10:29.919231 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.914947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:36821 (local address 127.14.75.3:40309)
0810 23:10:29.915360 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.915372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.915460 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.915708 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.915711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.915729 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.915854 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.915860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.916703 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.916706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.917539 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.917549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.918387 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.918390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.918533 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.918554 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.919004 (+ 450us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.919101 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":23,"thread_start_us":153,"threads_started":1}
I20260810 23:10:29.919481 3197 leader_election.cc:288] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96f6aa4fe5154919bed78e17d3f42bb4 (127.14.75.3:36721), 8b32fb1a3fbd4d3ea6ee575be1f2a11c (127.14.75.2:45399)
W20260810 23:10:29.919664 3174 tablet.cc:2369] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:10:29.920037 2989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" is_pre_election: true
I20260810 23:10:29.920351 2989 raft_consensus.cc:2454] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in term 0.
I20260810 23:10:29.920826 3059 leader_election.cc:302] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b32fb1a3fbd4d3ea6ee575be1f2a11c, 96f6aa4fe5154919bed78e17d3f42bb4; no voters:
I20260810 23:10:29.920886 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679aa5f3c18149b29c9f80130e9bb36e" is_pre_election: true
I20260810 23:10:29.921106 2852 raft_consensus.cc:2454] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in term 0.
I20260810 23:10:29.921116 3199 raft_consensus.cc:2781] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:10:29.921200 3199 raft_consensus.cc:482] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:10:29.921257 3199 raft_consensus.cc:3037] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:29.922566 3199 raft_consensus.cc:504] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.922950 3199 leader_election.cc:288] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [CANDIDATE]: Term 1 election: Requested vote from peers 8b32fb1a3fbd4d3ea6ee575be1f2a11c (127.14.75.2:45399), 679aa5f3c18149b29c9f80130e9bb36e (127.14.75.1:36821)
I20260810 23:10:29.923229 2989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c"
I20260810 23:10:29.923341 2989 raft_consensus.cc:3037] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:29.923478 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679aa5f3c18149b29c9f80130e9bb36e"
I20260810 23:10:29.923599 2852 raft_consensus.cc:3037] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:29.923820 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.919281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:53133 (local address 127.14.75.3:36721)
0810 23:10:29.919385 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.919389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.919592 (+ 203us) server_negotiation.cc:407] Connection header received
0810 23:10:29.919832 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.919835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.919879 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.919956 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.920297 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.920937 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.922531 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.923101 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.923197 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.923260 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.923612 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.923688 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:10:29.924226 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.919182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.3:36721 (local address 127.14.75.1:53133)
0810 23:10:29.919512 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.919525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.919685 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.919988 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.919990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.920007 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.920124 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.920131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.921182 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.921187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.922394 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.922408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.923329 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.923334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.923458 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.923480 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.923985 (+ 505us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.924084 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":96,"threads_started":1}
I20260810 23:10:29.924635 2989 raft_consensus.cc:2454] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in term 1.
I20260810 23:10:29.924793 2852 raft_consensus.cc:2454] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in term 1.
I20260810 23:10:29.925033 3059 leader_election.cc:302] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b32fb1a3fbd4d3ea6ee575be1f2a11c, 96f6aa4fe5154919bed78e17d3f42bb4; no voters:
I20260810 23:10:29.925184 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.920524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.75.1:56895 (local address 127.14.75.2:45399)
0810 23:10:29.921087 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.921092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.921488 (+ 396us) server_negotiation.cc:407] Connection header received
0810 23:10:29.921656 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.921659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.921702 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.921749 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:10:29.922085 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.922579 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.923905 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.924484 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.924579 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.924645 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.925009 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.925081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260810 23:10:29.925242 3125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "679aa5f3c18149b29c9f80130e9bb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" is_pre_election: true
I20260810 23:10:29.925498 3125 raft_consensus.cc:2379] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 679aa5f3c18149b29c9f80130e9bb36e in current term 1: Already voted for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in this term.
I20260810 23:10:29.925566 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.920447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.2:45399 (local address 127.14.75.1:56895)
0810 23:10:29.921395 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.921429 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:29.921548 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:29.921869 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:29.921871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:29.921884 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:10:29.921980 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.921983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.922700 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.922703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.923763 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:29.923774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.924750 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.924754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:29.924862 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.924887 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:10:29.925327 (+ 440us) client_negotiation.cc:240] Negotiation successful
0810 23:10:29.925400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":58,"threads_started":1}
I20260810 23:10:29.925892 3199 raft_consensus.cc:2781] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:29.925956 2989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eae7c5ac0a0d40a38a14c3bb810d2166" candidate_uuid: "679aa5f3c18149b29c9f80130e9bb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" is_pre_election: true
I20260810 23:10:29.926128 2989 raft_consensus.cc:2379] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 679aa5f3c18149b29c9f80130e9bb36e in current term 1: Already voted for candidate 96f6aa4fe5154919bed78e17d3f42bb4 in this term.
I20260810 23:10:29.926115 3199 raft_consensus.cc:686] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 LEADER]: Becoming Leader. State: Replica: 96f6aa4fe5154919bed78e17d3f42bb4, State: Running, Role: LEADER
I20260810 23:10:29.926429 3199 consensus_queue.cc:237] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } }
I20260810 23:10:29.926602 2786 leader_election.cc:302] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 679aa5f3c18149b29c9f80130e9bb36e; no voters: 8b32fb1a3fbd4d3ea6ee575be1f2a11c, 96f6aa4fe5154919bed78e17d3f42bb4
I20260810 23:10:29.926839 3197 raft_consensus.cc:2726] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:10:29.929347 2700 catalog_manager.cc:5495] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f6aa4fe5154919bed78e17d3f42bb4 (127.14.75.3). New cstate: current_term: 1 leader_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96f6aa4fe5154919bed78e17d3f42bb4" member_type: VOTER last_known_addr { host: "127.14.75.3" port: 36721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 } health_report { overall_health: UNKNOWN } } }
I20260810 23:10:29.956211 3187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.952904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.14.75.3:36721)
0810 23:10:29.953027 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:29.953031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:29.953047 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:10:29.953204 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:29.953206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:29.953265 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:29.953352 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:29.953601 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.954092 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.954903 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:29.955109 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:29.955184 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:29.955450 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:29.955873 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:29.955964 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:29.956030 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 23:10:29.956073 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:10:29.962163 2989 raft_consensus.cc:1261] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 1 FOLLOWER]: Refusing update from remote peer 96f6aa4fe5154919bed78e17d3f42bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:29.962163 2852 raft_consensus.cc:1261] T eae7c5ac0a0d40a38a14c3bb810d2166 P 679aa5f3c18149b29c9f80130e9bb36e [term 1 FOLLOWER]: Refusing update from remote peer 96f6aa4fe5154919bed78e17d3f42bb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:29.962877 3199 consensus_queue.cc:1035] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b32fb1a3fbd4d3ea6ee575be1f2a11c" member_type: VOTER last_known_addr { host: "127.14.75.2" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:29.963042 3203 consensus_queue.cc:1035] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "679aa5f3c18149b29c9f80130e9bb36e" member_type: VOTER last_known_addr { host: "127.14.75.1" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:29.979413 2700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:29.979606 2700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:10:29.982019 2700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f496eb88edb34673a7af9aad04a7d04d: Sending DeleteTablet for 3 replicas of tablet eae7c5ac0a0d40a38a14c3bb810d2166
I20260810 23:10:29.982776 3104 tablet_service.cc:1514] Processing DeleteTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:29 UTC) from {username='slave'} at 127.0.0.1:59140
I20260810 23:10:29.982960 2968 tablet_service.cc:1514] Processing DeleteTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:29 UTC) from {username='slave'} at 127.0.0.1:47046
I20260810 23:10:29.983341 2831 tablet_service.cc:1514] Processing DeleteTablet for tablet eae7c5ac0a0d40a38a14c3bb810d2166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:10:29 UTC) from {username='slave'} at 127.0.0.1:39470
I20260810 23:10:29.983417 3217 tablet_replica.cc:330] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: stopping tablet replica
I20260810 23:10:29.983733 3217 raft_consensus.cc:2229] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:10:29.983788 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2766
I20260810 23:10:29.983959 3216 tablet_replica.cc:330] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: stopping tablet replica
I20260810 23:10:29.984087 3217 raft_consensus.cc:2258] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:29.984392 3216 raft_consensus.cc:2229] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:10:29.984808 3216 raft_consensus.cc:2258] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:10:29.985764 3217 ts_tablet_manager.cc:1905] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:29.986222 3216 ts_tablet_manager.cc:1905] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:10:29.988610 3217 ts_tablet_manager.cc:1918] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:29.988706 3217 log.cc:1196] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-1/wal/wals/eae7c5ac0a0d40a38a14c3bb810d2166
I20260810 23:10:29.988895 3216 ts_tablet_manager.cc:1918] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:10:29.988983 3216 log.cc:1196] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Deleting WAL directory at /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786403171263156-14636-0/minicluster-data/ts-2/wal/wals/eae7c5ac0a0d40a38a14c3bb810d2166
I20260810 23:10:29.989045 3217 ts_tablet_manager.cc:1939] T eae7c5ac0a0d40a38a14c3bb810d2166 P 8b32fb1a3fbd4d3ea6ee575be1f2a11c: Deleting consensus metadata
I20260810 23:10:29.989289 3216 ts_tablet_manager.cc:1939] T eae7c5ac0a0d40a38a14c3bb810d2166 P 96f6aa4fe5154919bed78e17d3f42bb4: Deleting consensus metadata
I20260810 23:10:29.990320 2687 catalog_manager.cc:4841] TS 96f6aa4fe5154919bed78e17d3f42bb4 (127.14.75.3:36721): tablet eae7c5ac0a0d40a38a14c3bb810d2166 (table test-table [id=247dc3f7a7364baa9e3b2fcc83310a40]) successfully deleted
I20260810 23:10:29.994170 2688 catalog_manager.cc:4841] TS 8b32fb1a3fbd4d3ea6ee575be1f2a11c (127.14.75.2:45399): tablet eae7c5ac0a0d40a38a14c3bb810d2166 (table test-table [id=247dc3f7a7364baa9e3b2fcc83310a40]) successfully deleted
I20260810 23:10:29.994273 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2905
W20260810 23:10:29.994446 2688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.75.1:36821 (error 108)
I20260810 23:10:29.996387 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:29.995922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.1:36821 (local address 127.0.0.1:39484)
0810 23:10:29.996197 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:29.996270 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.1:36821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260810 23:10:29.996594 2688 catalog_manager.cc:4584] TS 679aa5f3c18149b29c9f80130e9bb36e (127.14.75.1:36821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eae7c5ac0a0d40a38a14c3bb810d2166: Network error: Client connection negotiation failed: client connection to 127.14.75.1:36821: connect: Connection refused (error 111)
I20260810 23:10:30.001153 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3040
I20260810 23:10:30.007917 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 2670
2026-08-10T23:10:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1214 ms)
[----------] 6 tests from EncryptionPolicyTest (11179 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:30 dist-test-slave-srgv krb5kdc[3222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:30 dist-test-slave-srgv krb5kdc[3222](info): set up 2 sockets
Aug 10 23:10:30 dist-test-slave-srgv krb5kdc[3222](info): commencing operation
krb5kdc: starting...
W20260810 23:10:31.070495 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:31.196985 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:31.309298 3236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:31.309698 3236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:31.309789 3236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:31.313935 3236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:31.314033 3236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:31.314073 3236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:31.314126 3236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:31.314162 3236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:31.319649 3236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:31.320756 3236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:31.321830 3236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:31.330349 3243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:31.330353 3241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:31.330467 3245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:31.330758 3236 server_base.cc:1034] running on GCE node
I20260810 23:10:31.331333 3236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:31.332347 3236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:31.333513 3236 hybrid_clock.cc:648] HybridClock initialized: now 1786403431333456 us; error 49 us; skew 500 ppm
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:31.336258 3236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:31.337159 3236 webserver.cc:460] Webserver started at http://127.0.0.1:35153/ using document root <none> and password file <none>
I20260810 23:10:31.337759 3236 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:31.337827 3236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:31.338003 3236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:31.340054 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "65ded71800d34c2587cd5d533a66dd6d"
format_stamp: "Formatted at 2026-08-10 23:10:31 on dist-test-slave-srgv"
server_key: "6b80a240392d36ba3b1deeb7c87dc7fa"
server_key_iv: "7c7cc9c4ffc3625eb11a7fccf6e8ad13"
server_key_version: "encryptionkey@0"
I20260810 23:10:31.340597 3236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "65ded71800d34c2587cd5d533a66dd6d"
format_stamp: "Formatted at 2026-08-10 23:10:31 on dist-test-slave-srgv"
server_key: "6b80a240392d36ba3b1deeb7c87dc7fa"
server_key_iv: "7c7cc9c4ffc3625eb11a7fccf6e8ad13"
server_key_version: "encryptionkey@0"
I20260810 23:10:31.344329 3236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 23:10:31.346665 3252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:31.347867 3236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:31.348024 3236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "65ded71800d34c2587cd5d533a66dd6d"
format_stamp: "Formatted at 2026-08-10 23:10:31 on dist-test-slave-srgv"
server_key: "6b80a240392d36ba3b1deeb7c87dc7fa"
server_key_iv: "7c7cc9c4ffc3625eb11a7fccf6e8ad13"
server_key_version: "encryptionkey@0"
I20260810 23:10:31.348138 3236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:31.366861 3236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:31.370003 3236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:31.370225 3236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:31.377640 3236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42667
I20260810 23:10:31.377704 3307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42667 every 8 connection(s)
I20260810 23:10:31.378664 3236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:31.381444 3308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:31.383216 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3236
I20260810 23:10:31.383365 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:31.383669 14636 external_mini_cluster.cc:1428] Setting key 41aa886a13071c901137c49de257edd0
I20260810 23:10:31.387917 3308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: Bootstrap starting.
I20260810 23:10:31.390412 3308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:31.391338 3308 log.cc:824] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:10:31.393437 3308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: No bootstrap required, opened a new log
I20260810 23:10:31.398274 3308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260810 23:10:31.398535 3308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:31.398593 3308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ded71800d34c2587cd5d533a66dd6d, State: Initialized, Role: FOLLOWER
I20260810 23:10:31.398692 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:31.385034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39681 (local address 127.0.0.1:42667)
0810 23:10:31.385468 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:31.385484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:31.385518 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:10:31.386138 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:31.386154 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:31.386479 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:31.386744 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:31.387663 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:31.388703 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:31.389916 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:31.390383 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:31.390558 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:31.393437 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:31.393447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:31.393466 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:31.393500 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:31.395422 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:31.396033 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:31.396040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:31.396047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:31.396186 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:31.396529 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:31.396533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:31.396534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:31.397024 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:31.397236 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:31.397507 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 23:10:31.397739 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260810 23:10:31.399080 3308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260810 23:10:31.399250 3308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:31.399345 3308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:31.399443 3308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:31.400602 3308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260810 23:10:31.400982 3308 leader_election.cc:302] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ded71800d34c2587cd5d533a66dd6d; no voters:
I20260810 23:10:31.401335 3308 leader_election.cc:288] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:31.401461 3313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:31.401687 3313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [term 1 LEADER]: Becoming Leader. State: Replica: 65ded71800d34c2587cd5d533a66dd6d, State: Running, Role: LEADER
I20260810 23:10:31.402062 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:31.402025 3313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } }
I20260810 23:10:31.402724 3308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:31.403214 3313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ded71800d34c2587cd5d533a66dd6d. Latest consensus state: current_term: 1 leader_uuid: "65ded71800d34c2587cd5d533a66dd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260810 23:10:31.403359 3313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:31.403805 3314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ded71800d34c2587cd5d533a66dd6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ded71800d34c2587cd5d533a66dd6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42667 } } }
I20260810 23:10:31.403972 3314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:31.407994 3334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:31.408084 3334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:31.408146 3322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:31.408993 3322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:31.414103 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:31.404563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40907 (local address 127.0.0.1:42667)
0810 23:10:31.404707 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:31.404713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:31.405745 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 23:10:31.405887 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:31.405890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:31.405935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:31.406031 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:31.406944 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:31.407618 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:31.408548 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:31.408781 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:31.408894 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:31.410493 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:31.410496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:31.410499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:31.410532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:31.412035 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:31.412525 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:31.412531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:31.412537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:31.412610 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:31.412959 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:31.412962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:31.412963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:31.413198 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:31.413368 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:31.413825 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 23:10:31.413923 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:10:31.415407 3322 catalog_manager.cc:1347] Generated new cluster ID: d05687563ead473089b3dc2aa6ba88e2
I20260810 23:10:31.415486 3322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:31.430124 3322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:31.431072 3322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:31.436507 3322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ded71800d34c2587cd5d533a66dd6d: Generated new TSK 0
I20260810 23:10:31.437259 3322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:31.442266 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3236
W20260810 23:10:31.448630 3220 connection.cc:535] client connection to 127.0.0.1:42667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42667): Connection reset by peer (error 104)
2026-08-10T23:10:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3341](info): set up 2 sockets
Aug 10 23:10:31 dist-test-slave-srgv krb5kdc[3341](info): commencing operation
krb5kdc: starting...
W20260810 23:10:32.519246 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:32 dist-test-slave-srgv krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:32.646445 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:32.760067 3355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:32.760396 3355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:32.760484 3355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:32.764758 3355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:32.764856 3355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:32.764896 3355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:32.764955 3355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:32.764994 3355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:32.770274 3355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:32.771396 3355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:32.772459 3355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:32.781133 3361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:32.781090 3363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:32.781091 3360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:32.781476 3355 server_base.cc:1034] running on GCE node
I20260810 23:10:32.782081 3355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:32.783361 3355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:32.784540 3355 hybrid_clock.cc:648] HybridClock initialized: now 1786403432784515 us; error 38 us; skew 500 ppm
Aug 10 23:10:32 dist-test-slave-srgv krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:32.787453 3355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:32.788467 3355 webserver.cc:460] Webserver started at http://127.0.0.1:39895/ using document root <none> and password file <none>
I20260810 23:10:32.789072 3355 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:32.789141 3355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:32.789455 3355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:32.791630 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "a2f683a2f0c04d3da9695db8d9b5eea7"
format_stamp: "Formatted at 2026-08-10 23:10:32 on dist-test-slave-srgv"
server_key: "87a792ce77da8c62624462862de1fe80"
server_key_iv: "bae8a8ca618ea72423e5a97d3ea15bdd"
server_key_version: "encryptionkey@0"
I20260810 23:10:32.792236 3355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "a2f683a2f0c04d3da9695db8d9b5eea7"
format_stamp: "Formatted at 2026-08-10 23:10:32 on dist-test-slave-srgv"
server_key: "87a792ce77da8c62624462862de1fe80"
server_key_iv: "bae8a8ca618ea72423e5a97d3ea15bdd"
server_key_version: "encryptionkey@0"
I20260810 23:10:32.796245 3355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:10:32.798832 3371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:32.800113 3355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:32.800256 3355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "a2f683a2f0c04d3da9695db8d9b5eea7"
format_stamp: "Formatted at 2026-08-10 23:10:32 on dist-test-slave-srgv"
server_key: "87a792ce77da8c62624462862de1fe80"
server_key_iv: "bae8a8ca618ea72423e5a97d3ea15bdd"
server_key_version: "encryptionkey@0"
I20260810 23:10:32.800388 3355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:32.816161 3355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:32.819373 3355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:32.819608 3355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:32.827220 3355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37037
I20260810 23:10:32.827247 3429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37037 every 8 connection(s)
I20260810 23:10:32.828269 3355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:32.831089 3430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:32.833676 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3355
I20260810 23:10:32.833841 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:32.834143 14636 external_mini_cluster.cc:1428] Setting key ad8db8e45df0a648486e48ac07cbd4aa
I20260810 23:10:32.837735 3430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7: Bootstrap starting.
I20260810 23:10:32.840463 3430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:32.841457 3430 log.cc:824] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:32.843931 3430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7: No bootstrap required, opened a new log
I20260810 23:10:32.844699 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:32.835668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44063 (local address 127.0.0.1:37037)
0810 23:10:32.836121 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:32.836140 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:32.836187 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:10:32.837019 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:32.837042 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:32.837517 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:32.837904 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:32.838354 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:32.839793 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:32.840788 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:32.841570 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:32.841728 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:32.842668 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:32.842674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:32.842691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:32.843195 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:32.843265 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:32.843297 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:32.843440 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:32.843570 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 23:10:32.843707 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
I20260810 23:10:32.847999 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:32.848405 3430 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260810 23:10:32.848721 3430 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:32.848780 3430 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f683a2f0c04d3da9695db8d9b5eea7, State: Initialized, Role: FOLLOWER
I20260810 23:10:32.849279 3430 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260810 23:10:32.849496 3430 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:32.849592 3430 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:32.849711 3430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:32.850872 3430 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260810 23:10:32.851240 3430 leader_election.cc:302] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f683a2f0c04d3da9695db8d9b5eea7; no voters:
I20260810 23:10:32.851601 3430 leader_election.cc:288] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:32.851742 3440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:32.852001 3440 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [term 1 LEADER]: Becoming Leader. State: Replica: a2f683a2f0c04d3da9695db8d9b5eea7, State: Running, Role: LEADER
I20260810 23:10:32.852392 3440 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } }
I20260810 23:10:32.852911 3430 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:32.853691 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f683a2f0c04d3da9695db8d9b5eea7. Latest consensus state: current_term: 1 leader_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } } }
I20260810 23:10:32.853940 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:32.854504 3449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:32.856006 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:32.850499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35025 (local address 127.0.0.1:37037)
0810 23:10:32.850638 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:32.850643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:32.850857 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:10:32.851013 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:32.851016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:32.851079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:32.851174 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:32.851486 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:32.852261 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:32.853487 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:32.853895 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:32.854017 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:32.855193 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:32.855197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:32.855200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:32.855215 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:32.855247 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:32.855262 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:32.855344 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:32.855813 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 23:10:32.855851 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:10:32.854802 3442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f683a2f0c04d3da9695db8d9b5eea7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37037 } } }
I20260810 23:10:32.856513 3442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:32.859000 3449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:32.865286 3449 catalog_manager.cc:1347] Generated new cluster ID: d969f71ffab4407db03d394a86db5234
I20260810 23:10:32.865449 3449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:32.881919 3449 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:32.882844 3449 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:32.892539 3449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f683a2f0c04d3da9695db8d9b5eea7: Generated new TSK 0
I20260810 23:10:32.893280 3449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:32.898394 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3355
W20260810 23:10:32.905537 3339 connection.cc:535] client connection to 127.0.0.1:37037 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37037): Connection reset by peer (error 104)
2026-08-10T23:10:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:32 dist-test-slave-srgv krb5kdc[3462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:32 dist-test-slave-srgv krb5kdc[3462](info): set up 2 sockets
Aug 10 23:10:32 dist-test-slave-srgv krb5kdc[3462](info): commencing operation
krb5kdc: starting...
W20260810 23:10:33.965901 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:34 dist-test-slave-srgv krb5kdc[3462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:34.092691 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36771
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:34.207584 3476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:34.207863 3476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:34.207907 3476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:34.212234 3476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:34.212309 3476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:34.212329 3476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:34.212374 3476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:34.212410 3476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:34.217968 3476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36771
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:34.219190 3476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:34.220232 3476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:34.229023 3482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:34.229040 3481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:34.229173 3476 server_base.cc:1034] running on GCE node
W20260810 23:10:34.229023 3484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:34.229876 3476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:34.231022 3476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:34.232192 3476 hybrid_clock.cc:648] HybridClock initialized: now 1786403434232165 us; error 40 us; skew 500 ppm
Aug 10 23:10:34 dist-test-slave-srgv krb5kdc[3462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:34.235370 3476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:34.236304 3476 webserver.cc:460] Webserver started at http://127.0.0.1:42935/ using document root <none> and password file <none>
I20260810 23:10:34.236891 3476 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:34.236955 3476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:34.237200 3476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:34.239424 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "76778328195f4fd697c0f757bd038c11"
format_stamp: "Formatted at 2026-08-10 23:10:34 on dist-test-slave-srgv"
server_key: "1f6572c3eeb56667ccee2d45b782d3c6"
server_key_iv: "c8d7e66fd57b3c1494665278b62a7f0a"
server_key_version: "encryptionkey@0"
I20260810 23:10:34.240026 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "76778328195f4fd697c0f757bd038c11"
format_stamp: "Formatted at 2026-08-10 23:10:34 on dist-test-slave-srgv"
server_key: "1f6572c3eeb56667ccee2d45b782d3c6"
server_key_iv: "c8d7e66fd57b3c1494665278b62a7f0a"
server_key_version: "encryptionkey@0"
I20260810 23:10:34.243918 3476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:10:34.246481 3491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:34.247861 3476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:10:34.247977 3476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "76778328195f4fd697c0f757bd038c11"
format_stamp: "Formatted at 2026-08-10 23:10:34 on dist-test-slave-srgv"
server_key: "1f6572c3eeb56667ccee2d45b782d3c6"
server_key_iv: "c8d7e66fd57b3c1494665278b62a7f0a"
server_key_version: "encryptionkey@0"
I20260810 23:10:34.248067 3476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:34.252629 3476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:34.255981 3476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:34.256142 3476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:34.262930 3476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36771
I20260810 23:10:34.262992 3546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36771 every 8 connection(s)
I20260810 23:10:34.263967 3476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:34.266877 3547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:34.269711 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3476
I20260810 23:10:34.269865 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:34.270161 14636 external_mini_cluster.cc:1428] Setting key 354f58e9c49f4c4de6c4076f9da8f9ec
I20260810 23:10:34.273134 3547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11: Bootstrap starting.
I20260810 23:10:34.275435 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:34.271623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36725 (local address 127.0.0.1:36771)
0810 23:10:34.272084 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:34.272099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:34.272136 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:10:34.272755 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:34.272772 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:34.273047 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:34.273393 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:34.274214 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:34.274218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:34.274229 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:34.274239 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:34.274296 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:34.274314 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:34.274427 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:34.274558 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:10:34.274723 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
I20260810 23:10:34.275760 3547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:34.276587 3547 log.cc:824] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:34.278393 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:34.278812 3547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11: No bootstrap required, opened a new log
I20260810 23:10:34.282403 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:34.280576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46421 (local address 127.0.0.1:36771)
0810 23:10:34.280713 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:34.280717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:34.280889 (+ 172us) server_negotiation.cc:407] Connection header received
0810 23:10:34.281006 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:34.281008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:34.281037 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:34.281104 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:34.281957 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:34.281960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:34.281962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:34.281973 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:34.281998 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:34.282009 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:34.282076 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:34.282215 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 23:10:34.282248 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:10:34.283170 3547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } }
I20260810 23:10:34.283432 3547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:34.283493 3547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76778328195f4fd697c0f757bd038c11, State: Initialized, Role: FOLLOWER
I20260810 23:10:34.283910 3547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } }
I20260810 23:10:34.284089 3547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:34.284181 3547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:34.284282 3547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:34.285264 3547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } }
I20260810 23:10:34.285665 3547 leader_election.cc:302] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76778328195f4fd697c0f757bd038c11; no voters:
I20260810 23:10:34.286007 3547 leader_election.cc:288] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:34.286151 3557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:34.286401 3557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [term 1 LEADER]: Becoming Leader. State: Replica: 76778328195f4fd697c0f757bd038c11, State: Running, Role: LEADER
I20260810 23:10:34.286806 3557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } }
I20260810 23:10:34.287369 3547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:34.288622 3558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76778328195f4fd697c0f757bd038c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } } }
I20260810 23:10:34.288749 3558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:34.289177 3566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:34.289162 3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76778328195f4fd697c0f757bd038c11. Latest consensus state: current_term: 1 leader_uuid: "76778328195f4fd697c0f757bd038c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76778328195f4fd697c0f757bd038c11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36771 } } }
I20260810 23:10:34.290010 3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:34.292541 3566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:34.298098 3566 catalog_manager.cc:1347] Generated new cluster ID: ce7aacde4e054884a47abd1d47157c6c
I20260810 23:10:34.298221 3566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:34.326648 3566 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:34.327052 3566 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:34.332278 3566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76778328195f4fd697c0f757bd038c11: Generated new TSK 0
I20260810 23:10:34.333021 3566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:34.343187 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3476
2026-08-10T23:10:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:34 dist-test-slave-srgv krb5kdc[3579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:34 dist-test-slave-srgv krb5kdc[3579](info): set up 2 sockets
Aug 10 23:10:34 dist-test-slave-srgv krb5kdc[3579](info): commencing operation
krb5kdc: starting...
W20260810 23:10:35.398695 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:35.523869 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:35.634305 3593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:35.634569 3593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:35.634611 3593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:35.638746 3593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:35.638808 3593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:35.638826 3593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:35.638844 3593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:35.638859 3593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:35.644057 3593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:35.644986 3593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:35.645923 3593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:35.654207 3598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:35.654331 3593 server_base.cc:1034] running on GCE node
W20260810 23:10:35.654203 3599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:35.654203 3602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:35.655100 3593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:35.656124 3593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:35.657313 3593 hybrid_clock.cc:648] HybridClock initialized: now 1786403435657291 us; error 47 us; skew 500 ppm
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:35.664239 3593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:35.665127 3593 webserver.cc:460] Webserver started at http://127.0.0.1:43695/ using document root <none> and password file <none>
I20260810 23:10:35.665709 3593 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:35.665779 3593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:35.665947 3593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:35.667907 3593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "d414494ee6974769ad6553d40f27f1cf"
format_stamp: "Formatted at 2026-08-10 23:10:35 on dist-test-slave-srgv"
server_key: "02ed5ca392a03ac07edf926149f3e69c"
server_key_iv: "045a5c2144e8197e41212e9cd8e00b19"
server_key_version: "encryptionkey@0"
I20260810 23:10:35.668406 3593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "d414494ee6974769ad6553d40f27f1cf"
format_stamp: "Formatted at 2026-08-10 23:10:35 on dist-test-slave-srgv"
server_key: "02ed5ca392a03ac07edf926149f3e69c"
server_key_iv: "045a5c2144e8197e41212e9cd8e00b19"
server_key_version: "encryptionkey@0"
I20260810 23:10:35.672017 3593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 23:10:35.674324 3609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:35.675500 3593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:35.675597 3593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "d414494ee6974769ad6553d40f27f1cf"
format_stamp: "Formatted at 2026-08-10 23:10:35 on dist-test-slave-srgv"
server_key: "02ed5ca392a03ac07edf926149f3e69c"
server_key_iv: "045a5c2144e8197e41212e9cd8e00b19"
server_key_version: "encryptionkey@0"
I20260810 23:10:35.675676 3593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:35.723835 3593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:35.726907 3593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:35.727074 3593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:35.734354 3593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43179
I20260810 23:10:35.734383 3666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43179 every 8 connection(s)
I20260810 23:10:35.735299 3593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:35.738026 3667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:35.740897 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3593
I20260810 23:10:35.741045 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:35.741325 14636 external_mini_cluster.cc:1428] Setting key 28c77689b88a10ea54f5b84b63d9ccb6
I20260810 23:10:35.743868 3667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf: Bootstrap starting.
I20260810 23:10:35.746086 3667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:35.746816 3667 log.cc:824] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:35.749017 3667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf: No bootstrap required, opened a new log
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:10:35.753835 3667 raft_consensus.cc:348] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260810 23:10:35.754125 3667 raft_consensus.cc:374] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:35.754213 3667 raft_consensus.cc:729] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d414494ee6974769ad6553d40f27f1cf, State: Initialized, Role: FOLLOWER
I20260810 23:10:35.754719 3667 consensus_queue.cc:260] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260810 23:10:35.754895 3667 raft_consensus.cc:388] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:35.754971 3667 raft_consensus.cc:482] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:35.755095 3667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:35.756103 3667 raft_consensus.cc:504] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260810 23:10:35.756330 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:35.742735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53553 (local address 127.0.0.1:43179)
0810 23:10:35.743164 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:35.743177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:35.743212 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:10:35.743817 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:35.743833 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:35.744231 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:35.744554 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:35.745379 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:35.746479 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:35.747502 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:35.748189 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:35.748365 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:35.751133 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:35.751139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:35.751155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:35.751186 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:35.753247 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:35.753729 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:35.753734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:35.753740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:35.753888 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:35.754197 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:35.754201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:35.754202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:35.754723 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:35.754944 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:35.755252 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:10:35.755554 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260810 23:10:35.756505 3667 leader_election.cc:302] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d414494ee6974769ad6553d40f27f1cf; no voters:
I20260810 23:10:35.756850 3667 leader_election.cc:288] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:35.756971 3673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:35.757238 3673 raft_consensus.cc:686] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [term 1 LEADER]: Becoming Leader. State: Replica: d414494ee6974769ad6553d40f27f1cf, State: Running, Role: LEADER
I20260810 23:10:35.757642 3673 consensus_queue.cc:237] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } }
I20260810 23:10:35.758112 3667 sys_catalog.cc:564] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:35.759233 3676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d414494ee6974769ad6553d40f27f1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } } }
I20260810 23:10:35.759377 3676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:35.759518 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:35.759608 3676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d414494ee6974769ad6553d40f27f1cf. Latest consensus state: current_term: 1 leader_uuid: "d414494ee6974769ad6553d40f27f1cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414494ee6974769ad6553d40f27f1cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43179 } } }
I20260810 23:10:35.759684 3676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:35.760866 3684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:35.763896 3684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:35.770466 3684 catalog_manager.cc:1347] Generated new cluster ID: f59dd882ecc74a2dbb3034c614923391
I20260810 23:10:35.770692 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:35.761983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37477 (local address 127.0.0.1:43179)
0810 23:10:35.762117 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:35.762124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:35.762373 (+ 249us) server_negotiation.cc:407] Connection header received
0810 23:10:35.762473 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:35.762476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:35.762522 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:35.762629 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:35.763433 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:35.764168 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:35.765326 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:35.765691 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:35.765809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:35.767214 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:35.767218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:35.767220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:35.767245 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:35.768879 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:35.769320 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:35.769325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:35.769327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:35.769466 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:35.769754 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:35.769757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:35.769759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:35.769983 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:35.770111 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:35.770411 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:10:35.770532 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:10:35.770761 3684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:35.788975 3684 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:35.790210 3684 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:35.796460 3684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d414494ee6974769ad6553d40f27f1cf: Generated new TSK 0
I20260810 23:10:35.797133 3684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:35.808096 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3593
2026-08-10T23:10:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3702](info): set up 2 sockets
Aug 10 23:10:35 dist-test-slave-srgv krb5kdc[3702](info): commencing operation
krb5kdc: starting...
W20260810 23:10:36.870368 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:36 dist-test-slave-srgv krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:36.996500 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:37.108292 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:37.108565 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:37.108609 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:37.112564 3717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:37.112618 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:37.112656 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:37.112672 3717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:37.112689 3717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:37.118000 3717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:37.118978 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:37.119936 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:37.128497 3726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:37.128521 3722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:37.128592 3724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:37.128803 3717 server_base.cc:1034] running on GCE node
I20260810 23:10:37.129495 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:37.130618 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:37.131807 3717 hybrid_clock.cc:648] HybridClock initialized: now 1786403437131778 us; error 48 us; skew 500 ppm
Aug 10 23:10:37 dist-test-slave-srgv krb5kdc[3702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:37.134800 3717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:37.135715 3717 webserver.cc:460] Webserver started at http://127.0.0.1:44859/ using document root <none> and password file <none>
I20260810 23:10:37.136277 3717 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:37.136373 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:37.136607 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:37.138715 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "8606bda0c4314c8e9af6b9dde13242c5"
format_stamp: "Formatted at 2026-08-10 23:10:37 on dist-test-slave-srgv"
server_key: "7e22ce8a55ab2d5499d451fb96945633"
server_key_iv: "a380378c3461d081c665e59252a66924"
server_key_version: "encryptionkey@0"
I20260810 23:10:37.139303 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "8606bda0c4314c8e9af6b9dde13242c5"
format_stamp: "Formatted at 2026-08-10 23:10:37 on dist-test-slave-srgv"
server_key: "7e22ce8a55ab2d5499d451fb96945633"
server_key_iv: "a380378c3461d081c665e59252a66924"
server_key_version: "encryptionkey@0"
I20260810 23:10:37.143141 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:10:37.145663 3733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:37.146906 3717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:37.147048 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "8606bda0c4314c8e9af6b9dde13242c5"
format_stamp: "Formatted at 2026-08-10 23:10:37 on dist-test-slave-srgv"
server_key: "7e22ce8a55ab2d5499d451fb96945633"
server_key_iv: "a380378c3461d081c665e59252a66924"
server_key_version: "encryptionkey@0"
I20260810 23:10:37.147163 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:37.181392 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:37.185021 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:37.185256 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:37.192761 3717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36065
I20260810 23:10:37.192797 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36065 every 8 connection(s)
I20260810 23:10:37.193801 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:37.196708 3789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:37.202272 3789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5: Bootstrap starting.
I20260810 23:10:37.203754 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3717
I20260810 23:10:37.203938 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:37.204254 14636 external_mini_cluster.cc:1428] Setting key 5408e4a07f81077eb3fe7bd1bcbe7c19
I20260810 23:10:37.204890 3789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:37.205813 3789 log.cc:824] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:37.207952 3789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5: No bootstrap required, opened a new log
I20260810 23:10:37.212309 3789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } }
I20260810 23:10:37.212615 3789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:37.212682 3789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8606bda0c4314c8e9af6b9dde13242c5, State: Initialized, Role: FOLLOWER
I20260810 23:10:37.213148 3789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } }
I20260810 23:10:37.213306 3789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:37.213376 3789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:37.213506 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:37.205392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37405 (local address 127.0.0.1:36065)
0810 23:10:37.205874 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:37.205913 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:37.205959 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:10:37.206810 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:37.206828 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:37.207153 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:37.207437 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:37.207831 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:37.208851 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:37.210090 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:37.210570 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:37.210689 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:37.211629 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:37.211634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:37.211646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:37.212034 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:37.212087 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:37.212109 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:37.212204 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:37.212369 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:10:37.212600 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260810 23:10:37.213505 3789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:37.214507 3789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } }
I20260810 23:10:37.214856 3789 leader_election.cc:302] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8606bda0c4314c8e9af6b9dde13242c5; no voters:
I20260810 23:10:37.215186 3789 leader_election.cc:288] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:37.215291 3794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:37.215562 3794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8606bda0c4314c8e9af6b9dde13242c5, State: Running, Role: LEADER
I20260810 23:10:37.215893 3794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } }
I20260810 23:10:37.216501 3789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:37.216722 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:37.217109 3794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8606bda0c4314c8e9af6b9dde13242c5. Latest consensus state: current_term: 1 leader_uuid: "8606bda0c4314c8e9af6b9dde13242c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } } }
I20260810 23:10:37.217211 3794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:37.217602 3796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8606bda0c4314c8e9af6b9dde13242c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8606bda0c4314c8e9af6b9dde13242c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36065 } } }
I20260810 23:10:37.217798 3796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:37.217720 3799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:37.221244 3799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:37.224647 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:37.219497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42851 (local address 127.0.0.1:36065)
0810 23:10:37.219597 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:37.219602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:37.220163 (+ 561us) server_negotiation.cc:407] Connection header received
0810 23:10:37.220273 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:37.220276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:37.220325 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:37.220414 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:37.220719 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:37.221523 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:37.222729 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:37.223063 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:37.223215 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:37.224194 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:37.224198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:37.224200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:37.224212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:37.224249 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:37.224262 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:37.224346 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:37.224439 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:10:37.224493 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:10:37.227412 3799 catalog_manager.cc:1347] Generated new cluster ID: b15c1cf712b74a0b8f21251e05917326
I20260810 23:10:37.227475 3799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:37.239634 3799 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:37.240545 3799 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:37.253213 3799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8606bda0c4314c8e9af6b9dde13242c5: Generated new TSK 0
I20260810 23:10:37.253962 3799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:37.258265 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3717
2026-08-10T23:10:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:37 dist-test-slave-srgv krb5kdc[3821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:37 dist-test-slave-srgv krb5kdc[3821](info): set up 2 sockets
Aug 10 23:10:37 dist-test-slave-srgv krb5kdc[3821](info): commencing operation
krb5kdc: starting...
W20260810 23:10:38.318485 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:38 dist-test-slave-srgv krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:38.444455 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:38.553287 3835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:38.553642 3835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:38.553736 3835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:38.558403 3835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:38.558503 3835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:38.558542 3835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:38.558581 3835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:38.558615 3835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:38.564098 3835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:38.565237 3835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:38.566278 3835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:38.574544 3842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:38.574548 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:38.574508 3841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:38.574813 3835 server_base.cc:1034] running on GCE node
I20260810 23:10:38.575330 3835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:38.576327 3835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:38.577479 3835 hybrid_clock.cc:648] HybridClock initialized: now 1786403438577455 us; error 37 us; skew 500 ppm
Aug 10 23:10:38 dist-test-slave-srgv krb5kdc[3821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:38.580189 3835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:38.581048 3835 webserver.cc:460] Webserver started at http://127.0.0.1:37529/ using document root <none> and password file <none>
I20260810 23:10:38.581645 3835 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:38.581710 3835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:38.581925 3835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:38.583871 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "775164d1db4642f7a8a054862cf547e3"
format_stamp: "Formatted at 2026-08-10 23:10:38 on dist-test-slave-srgv"
server_key: "4975e24367fbb7a6d87cfbab5f5c752e"
server_key_iv: "01b84b0f8b31127e3dd88b790b12662a"
server_key_version: "encryptionkey@0"
I20260810 23:10:38.584432 3835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "775164d1db4642f7a8a054862cf547e3"
format_stamp: "Formatted at 2026-08-10 23:10:38 on dist-test-slave-srgv"
server_key: "4975e24367fbb7a6d87cfbab5f5c752e"
server_key_iv: "01b84b0f8b31127e3dd88b790b12662a"
server_key_version: "encryptionkey@0"
I20260810 23:10:38.587953 3835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:10:38.590212 3851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:38.591372 3835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:38.591511 3835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "775164d1db4642f7a8a054862cf547e3"
format_stamp: "Formatted at 2026-08-10 23:10:38 on dist-test-slave-srgv"
server_key: "4975e24367fbb7a6d87cfbab5f5c752e"
server_key_iv: "01b84b0f8b31127e3dd88b790b12662a"
server_key_version: "encryptionkey@0"
I20260810 23:10:38.591629 3835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:38.596087 3835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:38.599057 3835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:38.599210 3835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:38.605895 3835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32867
I20260810 23:10:38.605978 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32867 every 8 connection(s)
I20260810 23:10:38.606869 3835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:38.609596 3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:38.610256 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3835
I20260810 23:10:38.610397 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:38.610678 14636 external_mini_cluster.cc:1428] Setting key 635fc8694dd19d8cf256d18175765f04
I20260810 23:10:38.616170 3904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3: Bootstrap starting.
I20260810 23:10:38.616273 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:38.612064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35715 (local address 127.0.0.1:32867)
0810 23:10:38.612508 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:38.612526 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:38.612567 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:10:38.613374 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:38.613400 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:38.613778 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:38.614165 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:38.614916 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:38.614922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:38.614939 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:38.614951 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:38.615019 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:38.615046 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:38.615196 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:38.615270 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:10:38.615413 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":109,"threads_started":1}
I20260810 23:10:38.618777 3904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:38.619104 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:38.619661 3904 log.cc:824] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:38.621797 3904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3: No bootstrap required, opened a new log
I20260810 23:10:38.622988 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:38.621383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46317 (local address 127.0.0.1:32867)
0810 23:10:38.621521 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:38.621526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:38.621711 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:10:38.621782 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:38.621785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:38.621822 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:38.621909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:38.622630 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:38.622634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:38.622637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:38.622648 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:38.622676 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:38.622689 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:38.622771 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:38.622842 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 23:10:38.622877 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:10:38.625463 3904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } }
I20260810 23:10:38.625695 3904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:38.625744 3904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775164d1db4642f7a8a054862cf547e3, State: Initialized, Role: FOLLOWER
I20260810 23:10:38.626091 3904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } }
I20260810 23:10:38.626192 3904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:38.626235 3904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:38.626302 3904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:38.627127 3904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } }
I20260810 23:10:38.627497 3904 leader_election.cc:302] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 775164d1db4642f7a8a054862cf547e3; no voters:
I20260810 23:10:38.627843 3904 leader_election.cc:288] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:38.627969 3914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:38.628202 3914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [term 1 LEADER]: Becoming Leader. State: Replica: 775164d1db4642f7a8a054862cf547e3, State: Running, Role: LEADER
I20260810 23:10:38.628485 3914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } }
I20260810 23:10:38.629016 3904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:38.630100 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "775164d1db4642f7a8a054862cf547e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } } }
I20260810 23:10:38.630231 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:38.630509 3918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 775164d1db4642f7a8a054862cf547e3. Latest consensus state: current_term: 1 leader_uuid: "775164d1db4642f7a8a054862cf547e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "775164d1db4642f7a8a054862cf547e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32867 } } }
I20260810 23:10:38.630591 3918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:38.630622 3925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:38.633559 3925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:38.638684 3925 catalog_manager.cc:1347] Generated new cluster ID: 88e317d24e264dca91e8bc26fd54a801
I20260810 23:10:38.638800 3925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:38.674242 3925 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:38.674592 3925 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:38.681450 3925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 775164d1db4642f7a8a054862cf547e3: Generated new TSK 0
I20260810 23:10:38.682200 3925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:38.692242 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3835
2026-08-10T23:10:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:38 dist-test-slave-srgv krb5kdc[3938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:38 dist-test-slave-srgv krb5kdc[3938](info): set up 2 sockets
Aug 10 23:10:38 dist-test-slave-srgv krb5kdc[3938](info): commencing operation
krb5kdc: starting...
W20260810 23:10:39.758354 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:39 dist-test-slave-srgv krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:39.889030 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:40.004896 3952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:40.005223 3952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:40.005316 3952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:40.009469 3952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:40.009572 3952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:40.009610 3952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:40.009647 3952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:40.009681 3952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:40.015313 3952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:40.016438 3952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:40.017632 3952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:40.026253 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:40.026288 3958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:40.026285 3959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:40.026568 3952 server_base.cc:1034] running on GCE node
I20260810 23:10:40.027204 3952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:40.028304 3952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:40.029549 3952 hybrid_clock.cc:648] HybridClock initialized: now 1786403440029525 us; error 30 us; skew 500 ppm
Aug 10 23:10:40 dist-test-slave-srgv krb5kdc[3938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:40.032490 3952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:40.033377 3952 webserver.cc:460] Webserver started at http://127.0.0.1:32971/ using document root <none> and password file <none>
I20260810 23:10:40.033999 3952 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:40.034096 3952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:40.034332 3952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:40.036423 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "2600ca242a344a36b0aa1b529505b95b"
format_stamp: "Formatted at 2026-08-10 23:10:40 on dist-test-slave-srgv"
server_key: "977ee82736000de85cf887acd5740e90"
server_key_iv: "e293bcb2e4ce914f4ab9e6311d2c4c02"
server_key_version: "encryptionkey@0"
I20260810 23:10:40.037003 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "2600ca242a344a36b0aa1b529505b95b"
format_stamp: "Formatted at 2026-08-10 23:10:40 on dist-test-slave-srgv"
server_key: "977ee82736000de85cf887acd5740e90"
server_key_iv: "e293bcb2e4ce914f4ab9e6311d2c4c02"
server_key_version: "encryptionkey@0"
I20260810 23:10:40.040743 3952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:10:40.043247 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:40.044524 3952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:10:40.044678 3952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "2600ca242a344a36b0aa1b529505b95b"
format_stamp: "Formatted at 2026-08-10 23:10:40 on dist-test-slave-srgv"
server_key: "977ee82736000de85cf887acd5740e90"
server_key_iv: "e293bcb2e4ce914f4ab9e6311d2c4c02"
server_key_version: "encryptionkey@0"
I20260810 23:10:40.044811 3952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:40.055702 3952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:40.059154 3952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:40.059378 3952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:40.066675 3952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44451
I20260810 23:10:40.066751 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44451 every 8 connection(s)
I20260810 23:10:40.067698 3952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:40.070520 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:40.076192 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 3952
I20260810 23:10:40.076239 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: Bootstrap starting.
I20260810 23:10:40.076362 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:40.076670 14636 external_mini_cluster.cc:1428] Setting key bd54c20d1c2a27c276d2ad86ff5e24ba
I20260810 23:10:40.079202 4027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:40.080112 4027 log.cc:824] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:40.082518 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: No bootstrap required, opened a new log
Aug 10 23:10:40 dist-test-slave-srgv krb5kdc[3938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:10:40.086781 4027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260810 23:10:40.087091 4027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:40.087167 4027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2600ca242a344a36b0aa1b529505b95b, State: Initialized, Role: FOLLOWER
I20260810 23:10:40.087680 4027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260810 23:10:40.087815 4027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:40.087909 4027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:40.088044 4027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:40.089004 4027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260810 23:10:40.089373 4027 leader_election.cc:302] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2600ca242a344a36b0aa1b529505b95b; no voters:
I20260810 23:10:40.089833 4027 leader_election.cc:288] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:40.089915 4032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:40.090122 4032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [term 1 LEADER]: Becoming Leader. State: Replica: 2600ca242a344a36b0aa1b529505b95b, State: Running, Role: LEADER
I20260810 23:10:40.090484 4032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260810 23:10:40.091223 4027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:40.092273 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2600ca242a344a36b0aa1b529505b95b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260810 23:10:40.092669 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:40.092571 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2600ca242a344a36b0aa1b529505b95b. Latest consensus state: current_term: 1 leader_uuid: "2600ca242a344a36b0aa1b529505b95b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2600ca242a344a36b0aa1b529505b95b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260810 23:10:40.093026 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:40.093608 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:40.078087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:45047 (local address 127.0.0.1:44451)
0810 23:10:40.078543 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:40.078562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:40.078605 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:10:40.079523 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:40.079547 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:40.080005 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:40.080427 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:40.081329 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:40.082805 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:40.083683 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:40.084438 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:40.084604 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:40.087586 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:40.087592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:40.087606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:40.087636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:40.089720 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:40.090371 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:40.090380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:40.090387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:40.090533 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:40.090947 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:40.090949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:40.090950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:40.091402 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:40.091642 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:40.092229 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 23:10:40.092507 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260810 23:10:40.097189 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 23:10:40.098246 4048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:40.098361 4048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:40.099200 4043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:40.100123 4043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:40.106823 4043 catalog_manager.cc:1347] Generated new cluster ID: 408bf9cc70604112969ec5f71a630a5b
I20260810 23:10:40.106985 4043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:40.108582 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:40.099510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:51089 (local address 127.0.0.1:44451)
0810 23:10:40.099629 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:40.099634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:40.099998 (+ 364us) server_negotiation.cc:407] Connection header received
0810 23:10:40.100151 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:40.100154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:40.100207 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:40.100282 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:40.100964 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:40.101586 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:40.102588 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:40.102965 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:40.103083 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:40.104832 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:40.104836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:40.104838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:40.104871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:40.106647 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:40.107239 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:40.107241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:40.107243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:40.107316 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:40.107681 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:40.107684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:40.107686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:40.107876 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:40.108042 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:40.108320 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 23:10:40.108422 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:10:40.130132 4043 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:40.131224 4043 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:40.138566 4043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2600ca242a344a36b0aa1b529505b95b: Generated new TSK 0
I20260810 23:10:40.139364 4043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:40.150857 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 3952
W20260810 23:10:40.157562 3936 connection.cc:535] client connection to 127.0.0.1:44451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44451): Connection reset by peer (error 104)
2026-08-10T23:10:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:40 dist-test-slave-srgv krb5kdc[4062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:40 dist-test-slave-srgv krb5kdc[4062](info): set up 2 sockets
Aug 10 23:10:40 dist-test-slave-srgv krb5kdc[4062](info): commencing operation
krb5kdc: starting...
W20260810 23:10:41.210305 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:41 dist-test-slave-srgv krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:41.334631 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:41.444802 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:41.445079 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:41.445123 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:41.449239 4075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:41.449319 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:41.449340 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:41.449357 4075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:41.449435 4075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:41.454815 4075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:41.455852 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:41.456859 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:41.465210 4082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:41.465232 4084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:41.465229 4081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:41.465374 4075 server_base.cc:1034] running on GCE node
I20260810 23:10:41.466125 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:41.467227 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:41.468429 4075 hybrid_clock.cc:648] HybridClock initialized: now 1786403441468422 us; error 32 us; skew 500 ppm
Aug 10 23:10:41 dist-test-slave-srgv krb5kdc[4062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:41.471215 4075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:41.472096 4075 webserver.cc:460] Webserver started at http://127.0.0.1:40457/ using document root <none> and password file <none>
I20260810 23:10:41.472651 4075 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:41.472755 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:41.472990 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:41.475044 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "7c455255507d4ef991ea6dcb226a9e4e"
format_stamp: "Formatted at 2026-08-10 23:10:41 on dist-test-slave-srgv"
server_key: "e18720377bc5ae9d8cace9340497f1e6"
server_key_iv: "26bcfeb74e94c8342a663b81f7c8de76"
server_key_version: "encryptionkey@0"
I20260810 23:10:41.475610 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "7c455255507d4ef991ea6dcb226a9e4e"
format_stamp: "Formatted at 2026-08-10 23:10:41 on dist-test-slave-srgv"
server_key: "e18720377bc5ae9d8cace9340497f1e6"
server_key_iv: "26bcfeb74e94c8342a663b81f7c8de76"
server_key_version: "encryptionkey@0"
I20260810 23:10:41.479295 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:10:41.481679 4091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:41.482905 4075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:10:41.483048 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "7c455255507d4ef991ea6dcb226a9e4e"
format_stamp: "Formatted at 2026-08-10 23:10:41 on dist-test-slave-srgv"
server_key: "e18720377bc5ae9d8cace9340497f1e6"
server_key_iv: "26bcfeb74e94c8342a663b81f7c8de76"
server_key_version: "encryptionkey@0"
I20260810 23:10:41.483160 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:41.493618 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:41.496379 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:41.496595 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:41.503509 4075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46795
I20260810 23:10:41.503585 4145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46795 every 8 connection(s)
I20260810 23:10:41.504498 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:41.507344 4146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:41.511101 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4075
I20260810 23:10:41.511265 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:41.511560 14636 external_mini_cluster.cc:1428] Setting key cbad0a1d51ef84b7a686c31e2ebddbcc
I20260810 23:10:41.513593 4146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: Bootstrap starting.
I20260810 23:10:41.516283 4146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:41.517131 4146 log.cc:824] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:41.519315 4146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: No bootstrap required, opened a new log
I20260810 23:10:41.521932 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:41.512944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:44377 (local address 127.0.0.1:46795)
0810 23:10:41.513385 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:41.513430 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:41.513475 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:10:41.514296 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:41.514320 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:41.514747 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:41.515120 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:41.515518 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:41.517010 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:41.517997 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:41.518677 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:41.518846 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:41.519687 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:41.519695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:41.519711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:41.520191 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:41.520256 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:41.520293 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:41.520423 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:41.520554 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:10:41.520696 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
I20260810 23:10:41.523123 4146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260810 23:10:41.523388 4146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:41.523447 4146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c455255507d4ef991ea6dcb226a9e4e, State: Initialized, Role: FOLLOWER
I20260810 23:10:41.523846 4146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260810 23:10:41.524039 4146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:41.524118 4146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:41.524235 4146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:41.524854 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:41.525337 4146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260810 23:10:41.525731 4146 leader_election.cc:302] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c455255507d4ef991ea6dcb226a9e4e; no voters:
I20260810 23:10:41.526098 4146 leader_election.cc:288] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:41.526199 4152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:41.526502 4152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [term 1 LEADER]: Becoming Leader. State: Replica: 7c455255507d4ef991ea6dcb226a9e4e, State: Running, Role: LEADER
I20260810 23:10:41.526857 4152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260810 23:10:41.527448 4146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:41.528007 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c455255507d4ef991ea6dcb226a9e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260810 23:10:41.528115 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:41.528311 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c455255507d4ef991ea6dcb226a9e4e. Latest consensus state: current_term: 1 leader_uuid: "7c455255507d4ef991ea6dcb226a9e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c455255507d4ef991ea6dcb226a9e4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260810 23:10:41.528403 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:41.532373 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:41.527471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:35427 (local address 127.0.0.1:46795)
0810 23:10:41.527578 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:41.527582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:41.527596 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:10:41.527789 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:41.527793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:41.527851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:41.527950 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:41.528471 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:41.529333 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:41.530571 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:41.530829 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:41.530950 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:41.531855 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:41.531858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:41.531861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:41.531874 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:41.531906 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:41.531917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:41.532017 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:41.532172 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:10:41.532208 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 23:10:41.532620 4175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:41.532706 4175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:41.532802 4161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:41.533660 4161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:41.539192 4161 catalog_manager.cc:1347] Generated new cluster ID: 79bcc27005fd4705b9b41223f0d16ca0
I20260810 23:10:41.539256 4161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:41.552846 4161 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:41.554018 4161 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:41.566463 4161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c455255507d4ef991ea6dcb226a9e4e: Generated new TSK 0
I20260810 23:10:41.567163 4161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:41.577888 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4075
W20260810 23:10:41.584548 4060 connection.cc:535] client connection to 127.0.0.1:46795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46795): Connection reset by peer (error 104)
2026-08-10T23:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:41 dist-test-slave-srgv krb5kdc[4180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:41 dist-test-slave-srgv krb5kdc[4180](info): set up 2 sockets
Aug 10 23:10:41 dist-test-slave-srgv krb5kdc[4180](info): commencing operation
krb5kdc: starting...
W20260810 23:10:42.646574 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:42 dist-test-slave-srgv krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:42.770511 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:42.881093 4193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:42.881399 4193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:42.881515 4193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:42.885532 4193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:42.885625 4193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:42.885663 4193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:42.885722 4193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:42.885762 4193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:42.890973 4193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:42.892051 4193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:42.893075 4193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:42.902429 4199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:42.902468 4198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:42.902429 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:42.902832 4193 server_base.cc:1034] running on GCE node
I20260810 23:10:42.903363 4193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:42.904388 4193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:42.905531 4193 hybrid_clock.cc:648] HybridClock initialized: now 1786403442905514 us; error 31 us; skew 500 ppm
Aug 10 23:10:42 dist-test-slave-srgv krb5kdc[4180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:42.908217 4193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:42.909125 4193 webserver.cc:460] Webserver started at http://127.0.0.1:38073/ using document root <none> and password file <none>
I20260810 23:10:42.909744 4193 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:42.909837 4193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:42.910076 4193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:42.912083 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "d3c80d16a1ef4a329379ed148f81b62d"
format_stamp: "Formatted at 2026-08-10 23:10:42 on dist-test-slave-srgv"
server_key: "74416ee803612a85f57ac8023cede5b4"
server_key_iv: "930545a5c94eda7d1f2522301c532470"
server_key_version: "encryptionkey@0"
I20260810 23:10:42.912597 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "d3c80d16a1ef4a329379ed148f81b62d"
format_stamp: "Formatted at 2026-08-10 23:10:42 on dist-test-slave-srgv"
server_key: "74416ee803612a85f57ac8023cede5b4"
server_key_iv: "930545a5c94eda7d1f2522301c532470"
server_key_version: "encryptionkey@0"
I20260810 23:10:42.916280 4193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 23:10:42.918576 4210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:42.919749 4193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:42.919872 4193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "d3c80d16a1ef4a329379ed148f81b62d"
format_stamp: "Formatted at 2026-08-10 23:10:42 on dist-test-slave-srgv"
server_key: "74416ee803612a85f57ac8023cede5b4"
server_key_iv: "930545a5c94eda7d1f2522301c532470"
server_key_version: "encryptionkey@0"
I20260810 23:10:42.919981 4193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:42.924613 4193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:42.927523 4193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:42.927716 4193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:42.934957 4193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37467
I20260810 23:10:42.934969 4264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37467 every 8 connection(s)
I20260810 23:10:42.935884 4193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:42.936473 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4193
I20260810 23:10:42.936652 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:42.936945 14636 external_mini_cluster.cc:1428] Setting key 5e6b44c2294b00afdf50e22816c7cf9e
I20260810 23:10:42.939085 4265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:42.942478 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:42.938350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:40767 (local address 127.0.0.1:37467)
0810 23:10:42.938788 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:42.938809 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:42.938853 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:10:42.939685 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:42.939706 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:42.940040 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:42.940400 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:42.941081 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:42.941087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:42.941101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:42.941113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:42.941174 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:42.941198 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:42.941349 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:42.941444 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:10:42.941597 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260810 23:10:42.945288 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:42.945840 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d: Bootstrap starting.
I20260810 23:10:42.948249 4265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:42.949172 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:42.947435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:37781 (local address 127.0.0.1:37467)
0810 23:10:42.947570 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:42.947575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:42.947779 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:10:42.947899 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:42.947901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:42.947931 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:42.947988 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:42.948738 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:42.948741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:42.948743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:42.948753 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:42.948786 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:42.948797 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:42.948866 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:42.948986 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:10:42.949020 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:10:42.949205 4265 log.cc:824] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:42.951126 4265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d: No bootstrap required, opened a new log
I20260810 23:10:42.954558 4265 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260810 23:10:42.954805 4265 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:42.954888 4265 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c80d16a1ef4a329379ed148f81b62d, State: Initialized, Role: FOLLOWER
I20260810 23:10:42.955340 4265 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260810 23:10:42.955492 4265 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:42.955561 4265 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:42.955660 4265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:42.956549 4265 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260810 23:10:42.956887 4265 leader_election.cc:302] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c80d16a1ef4a329379ed148f81b62d; no voters:
I20260810 23:10:42.957202 4265 leader_election.cc:288] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:42.957341 4275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:42.957629 4275 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [term 1 LEADER]: Becoming Leader. State: Replica: d3c80d16a1ef4a329379ed148f81b62d, State: Running, Role: LEADER
I20260810 23:10:42.957989 4275 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260810 23:10:42.958446 4265 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:42.959594 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c80d16a1ef4a329379ed148f81b62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260810 23:10:42.959986 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:42.959869 4277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c80d16a1ef4a329379ed148f81b62d. Latest consensus state: current_term: 1 leader_uuid: "d3c80d16a1ef4a329379ed148f81b62d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c80d16a1ef4a329379ed148f81b62d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260810 23:10:42.960357 4277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:42.960441 4290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:42.963279 4290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:42.968319 4290 catalog_manager.cc:1347] Generated new cluster ID: fa8d55b7de7b40dc96f6aaebaae6eced
I20260810 23:10:42.968389 4290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:42.987387 4290 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:42.987728 4290 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:42.998188 4290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c80d16a1ef4a329379ed148f81b62d: Generated new TSK 0
I20260810 23:10:42.998888 4290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:43.005877 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4193
2026-08-10T23:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:43 dist-test-slave-srgv krb5kdc[4299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:43 dist-test-slave-srgv krb5kdc[4299](info): set up 2 sockets
Aug 10 23:10:43 dist-test-slave-srgv krb5kdc[4299](info): commencing operation
krb5kdc: starting...
W20260810 23:10:44.070775 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:44.194797 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:44.303900 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:44.304191 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:44.304235 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:44.308115 4312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:44.308169 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:44.308188 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:44.308204 4312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:44.308218 4312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:44.313243 4312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:44.314325 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:44.315258 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:44.323671 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:44.323695 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:44.323704 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:44.323820 4312 server_base.cc:1034] running on GCE node
I20260810 23:10:44.324580 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:44.325631 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:44.326784 4312 hybrid_clock.cc:648] HybridClock initialized: now 1786403444326757 us; error 42 us; skew 500 ppm
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:44.329535 4312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:44.330417 4312 webserver.cc:460] Webserver started at http://127.0.0.1:37817/ using document root <none> and password file <none>
I20260810 23:10:44.330957 4312 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:44.331048 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:44.331265 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:44.333246 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9"
format_stamp: "Formatted at 2026-08-10 23:10:44 on dist-test-slave-srgv"
server_key: "e124462fc7a03fa0e8e7a59cb3fb3811"
server_key_iv: "7399ef07ae42d445787a87b2919a9c90"
server_key_version: "encryptionkey@0"
I20260810 23:10:44.333863 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9"
format_stamp: "Formatted at 2026-08-10 23:10:44 on dist-test-slave-srgv"
server_key: "e124462fc7a03fa0e8e7a59cb3fb3811"
server_key_iv: "7399ef07ae42d445787a87b2919a9c90"
server_key_version: "encryptionkey@0"
I20260810 23:10:44.337571 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:10:44.339797 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:44.340927 4312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:44.341076 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9"
format_stamp: "Formatted at 2026-08-10 23:10:44 on dist-test-slave-srgv"
server_key: "e124462fc7a03fa0e8e7a59cb3fb3811"
server_key_iv: "7399ef07ae42d445787a87b2919a9c90"
server_key_version: "encryptionkey@0"
I20260810 23:10:44.341190 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:44.367257 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:44.370162 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:44.370391 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:44.377787 4312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45951
I20260810 23:10:44.377812 4381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45951 every 8 connection(s)
I20260810 23:10:44.378705 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:44.381491 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:44.381620 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4312
I20260810 23:10:44.381750 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:44.382055 14636 external_mini_cluster.cc:1428] Setting key cb0e6c05ed8a158ac2cd8fb699d1123b
I20260810 23:10:44.388286 4382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9: Bootstrap starting.
I20260810 23:10:44.391108 4382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9: Neither blocks nor log segments found. Creating new log.
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:10:44.391855 4382 log.cc:824] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:44.393793 4382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9: No bootstrap required, opened a new log
I20260810 23:10:44.396512 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:44.383313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:52519 (local address 127.0.0.1:45951)
0810 23:10:44.383735 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:44.383764 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:44.383799 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:10:44.384659 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:44.384678 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:44.385007 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:44.385288 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:44.386242 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:44.387221 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:44.388353 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:44.388853 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:44.388967 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:44.391865 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:44.391870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:44.391881 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:44.391907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:44.393790 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:44.394341 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:44.394346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:44.394350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:44.394452 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:44.394797 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:44.394800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:44.394802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:44.395165 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:44.395380 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:44.395665 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:10:44.395865 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":163,"threads_started":1}
I20260810 23:10:44.397228 4382 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260810 23:10:44.397519 4382 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:44.397583 4382 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3206e1cc43f47c5b3d7becdfb56e3e9, State: Initialized, Role: FOLLOWER
I20260810 23:10:44.398008 4382 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260810 23:10:44.398169 4382 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:44.398262 4382 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:44.398370 4382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:44.399473 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:44.399590 4382 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260810 23:10:44.399922 4382 leader_election.cc:302] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3206e1cc43f47c5b3d7becdfb56e3e9; no voters:
I20260810 23:10:44.400280 4382 leader_election.cc:288] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:44.400386 4387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:44.400589 4387 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [term 1 LEADER]: Becoming Leader. State: Replica: d3206e1cc43f47c5b3d7becdfb56e3e9, State: Running, Role: LEADER
I20260810 23:10:44.400925 4387 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } }
I20260810 23:10:44.401496 4382 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:44.402464 4389 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3206e1cc43f47c5b3d7becdfb56e3e9. Latest consensus state: current_term: 1 leader_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } } }
I20260810 23:10:44.402611 4389 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:44.402827 4388 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3206e1cc43f47c5b3d7becdfb56e3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45951 } } }
I20260810 23:10:44.402928 4388 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:44.403015 4401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:44.406152 4401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:44.410995 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:44.401669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:46997 (local address 127.0.0.1:45951)
0810 23:10:44.401777 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:44.401781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:44.402029 (+ 248us) server_negotiation.cc:407] Connection header received
0810 23:10:44.402216 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:44.402219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:44.402264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:44.402342 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:44.403275 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:44.403978 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:44.405102 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:44.405357 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:44.405517 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:44.407146 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:44.407150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:44.407152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:44.407178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:44.408654 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:44.409243 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:44.409247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:44.409248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:44.409334 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:44.409764 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:44.409767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:44.409769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:44.409970 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:44.410120 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:44.410736 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 23:10:44.410840 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:10:44.412894 4401 catalog_manager.cc:1347] Generated new cluster ID: c26c34edc1eb47f39445f7b7dae6fac5
I20260810 23:10:44.412963 4401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:44.445667 4401 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:44.446614 4401 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:44.451081 4401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3206e1cc43f47c5b3d7becdfb56e3e9: Generated new TSK 0
I20260810 23:10:44.451889 4401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:44.456395 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4312
W20260810 23:10:44.462990 4297 connection.cc:535] client connection to 127.0.0.1:45951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45951): Connection reset by peer (error 104)
2026-08-10T23:10:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4413](info): set up 2 sockets
Aug 10 23:10:44 dist-test-slave-srgv krb5kdc[4413](info): commencing operation
krb5kdc: starting...
W20260810 23:10:45.518555 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:45 dist-test-slave-srgv krb5kdc[4413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:45.645002 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:45.755483 4428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:45.755808 4428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:45.755893 4428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:45.760018 4428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:45.760111 4428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:45.760150 4428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:45.760208 4428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:45.760245 4428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:45.766095 4428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:45.767242 4428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:45.768288 4428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:45.777168 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:45.777352 4428 server_base.cc:1034] running on GCE node
W20260810 23:10:45.777242 4433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:45.777172 4434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:45.778137 4428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:45.779294 4428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:45.780491 4428 hybrid_clock.cc:648] HybridClock initialized: now 1786403445780473 us; error 33 us; skew 500 ppm
Aug 10 23:10:45 dist-test-slave-srgv krb5kdc[4413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:45.783437 4428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:45.784390 4428 webserver.cc:460] Webserver started at http://127.0.0.1:41039/ using document root <none> and password file <none>
I20260810 23:10:45.784941 4428 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:45.785002 4428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:45.785173 4428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:45.787345 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "d30c8862c0dc4cb08cd87642dd3a7e20"
format_stamp: "Formatted at 2026-08-10 23:10:45 on dist-test-slave-srgv"
server_key: "be33e3cbaf92a5febed9ef57b4dcd4e7"
server_key_iv: "3cb67f1d436a26b52d612bccdb44ecf9"
server_key_version: "encryptionkey@0"
I20260810 23:10:45.787873 4428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "d30c8862c0dc4cb08cd87642dd3a7e20"
format_stamp: "Formatted at 2026-08-10 23:10:45 on dist-test-slave-srgv"
server_key: "be33e3cbaf92a5febed9ef57b4dcd4e7"
server_key_iv: "3cb67f1d436a26b52d612bccdb44ecf9"
server_key_version: "encryptionkey@0"
I20260810 23:10:45.791599 4428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:10:45.793999 4443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:45.795209 4428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:45.795306 4428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "d30c8862c0dc4cb08cd87642dd3a7e20"
format_stamp: "Formatted at 2026-08-10 23:10:45 on dist-test-slave-srgv"
server_key: "be33e3cbaf92a5febed9ef57b4dcd4e7"
server_key_iv: "3cb67f1d436a26b52d612bccdb44ecf9"
server_key_version: "encryptionkey@0"
I20260810 23:10:45.795387 4428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:45.810927 4428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:45.814296 4428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:45.814455 4428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:45.821936 4428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33585
I20260810 23:10:45.822017 4496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33585 every 8 connection(s)
I20260810 23:10:45.822901 4428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:45.825681 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:45.831256 4497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: Bootstrap starting.
I20260810 23:10:45.832086 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4428
I20260810 23:10:45.832221 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:45.832504 14636 external_mini_cluster.cc:1428] Setting key 9419c9e185b88fd494f3c57d9ef6fecd
I20260810 23:10:45.834116 4497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:45.834854 4497 log.cc:824] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:45.836720 4497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: No bootstrap required, opened a new log
I20260810 23:10:45.840289 4497 raft_consensus.cc:348] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260810 23:10:45.840643 4497 raft_consensus.cc:374] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:45.840780 4497 raft_consensus.cc:729] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d30c8862c0dc4cb08cd87642dd3a7e20, State: Initialized, Role: FOLLOWER
I20260810 23:10:45.841235 4497 consensus_queue.cc:260] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260810 23:10:45.841392 4497 raft_consensus.cc:388] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:45.841409 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:45.833622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:58959 (local address 127.0.0.1:33585)
0810 23:10:45.834049 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:45.834064 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:45.834099 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:10:45.834764 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:45.834780 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:45.835158 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:45.835490 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:45.835893 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:45.836866 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:45.837849 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:45.838399 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:45.838574 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:45.839500 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:45.839504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:45.839517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:45.839904 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:45.839956 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:45.839978 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:45.840119 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:45.840305 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 23:10:45.840449 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
I20260810 23:10:45.841488 4497 raft_consensus.cc:482] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:45.841599 4497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:45.842751 4497 raft_consensus.cc:504] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260810 23:10:45.843102 4497 leader_election.cc:302] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d30c8862c0dc4cb08cd87642dd3a7e20; no voters:
I20260810 23:10:45.843485 4497 leader_election.cc:288] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:45.843597 4502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:45.843847 4502 raft_consensus.cc:686] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [term 1 LEADER]: Becoming Leader. State: Replica: d30c8862c0dc4cb08cd87642dd3a7e20, State: Running, Role: LEADER
I20260810 23:10:45.844206 4502 consensus_queue.cc:237] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } }
I20260810 23:10:45.844614 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:45.844841 4497 sys_catalog.cc:564] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:45.846051 4503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } } }
I20260810 23:10:45.846203 4503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:45.846540 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d30c8862c0dc4cb08cd87642dd3a7e20. Latest consensus state: current_term: 1 leader_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d30c8862c0dc4cb08cd87642dd3a7e20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33585 } } }
I20260810 23:10:45.846649 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20 [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:45.850037 4521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:45.850134 4521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:45.850205 4516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:45.851071 4516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:45.852560 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:45.847337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:56125 (local address 127.0.0.1:33585)
0810 23:10:45.847449 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:45.847453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:45.847806 (+ 353us) server_negotiation.cc:407] Connection header received
0810 23:10:45.848003 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:45.848007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:45.848062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:45.848121 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:45.848607 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:45.849366 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:45.850571 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:45.850895 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:45.851006 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:45.851974 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:45.851977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:45.851980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:45.851994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:45.852022 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:45.852034 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:45.852170 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:45.852392 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:10:45.852429 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:10:45.856634 4516 catalog_manager.cc:1347] Generated new cluster ID: 9a0dbf5ba80548a4bc6b899d66b87653
I20260810 23:10:45.856699 4516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:45.878674 4516 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:45.880007 4516 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:45.887105 4516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d30c8862c0dc4cb08cd87642dd3a7e20: Generated new TSK 0
I20260810 23:10:45.887915 4516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:45.891642 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4428
W20260810 23:10:45.898804 4411 connection.cc:535] client connection to 127.0.0.1:33585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33585): Connection reset by peer (error 104)
2026-08-10T23:10:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:45 dist-test-slave-srgv krb5kdc[4528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:45 dist-test-slave-srgv krb5kdc[4528](info): set up 2 sockets
Aug 10 23:10:45 dist-test-slave-srgv krb5kdc[4528](info): commencing operation
krb5kdc: starting...
W20260810 23:10:46.967351 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:47 dist-test-slave-srgv krb5kdc[4528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:10:47.092397 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:47.204046 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:47.204378 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:47.204456 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:47.208576 4541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:47.208667 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:47.208706 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:47.208767 4541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:47.208804 4541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:47.214771 4541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:47.215896 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:47.216918 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:47.225415 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:47.225371 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:47.225524 4541 server_base.cc:1034] running on GCE node
W20260810 23:10:47.225385 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:47.226315 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:47.227340 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:47.228498 4541 hybrid_clock.cc:648] HybridClock initialized: now 1786403447228480 us; error 31 us; skew 500 ppm
Aug 10 23:10:47 dist-test-slave-srgv krb5kdc[4528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:47.235662 4541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:10:47.236627 4541 webserver.cc:460] Webserver started at http://127.0.0.1:33267/ using document root <none> and password file <none>
I20260810 23:10:47.237191 4541 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:47.237283 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:47.237571 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:47.239646 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "fc8ae625d640499d902e1abf3e17455d"
format_stamp: "Formatted at 2026-08-10 23:10:47 on dist-test-slave-srgv"
server_key: "0076654eeab851ec0d54fd5b0b98faf8"
server_key_iv: "92949e6de5ca2337fb7e32b5697478b1"
server_key_version: "encryptionkey@0"
I20260810 23:10:47.240232 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "fc8ae625d640499d902e1abf3e17455d"
format_stamp: "Formatted at 2026-08-10 23:10:47 on dist-test-slave-srgv"
server_key: "0076654eeab851ec0d54fd5b0b98faf8"
server_key_iv: "92949e6de5ca2337fb7e32b5697478b1"
server_key_version: "encryptionkey@0"
I20260810 23:10:47.244113 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:10:47.246518 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:47.247700 4541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:47.247843 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "fc8ae625d640499d902e1abf3e17455d"
format_stamp: "Formatted at 2026-08-10 23:10:47 on dist-test-slave-srgv"
server_key: "0076654eeab851ec0d54fd5b0b98faf8"
server_key_iv: "92949e6de5ca2337fb7e32b5697478b1"
server_key_version: "encryptionkey@0"
I20260810 23:10:47.247953 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:47.252663 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:47.256011 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:47.256212 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:47.263437 4541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42541
I20260810 23:10:47.263471 4610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42541 every 8 connection(s)
I20260810 23:10:47.264420 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:47.267169 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:47.268714 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4541
I20260810 23:10:47.268877 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:47.269166 14636 external_mini_cluster.cc:1428] Setting key 2a5c4f64c0927bc6277ed77121b2d0d2
I20260810 23:10:47.273344 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d: Bootstrap starting.
I20260810 23:10:47.274286 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:47.270606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:43507 (local address 127.0.0.1:42541)
0810 23:10:47.271057 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:47.271072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:47.271106 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:10:47.271743 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:47.271759 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:47.272024 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:47.272359 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:47.273140 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:47.273145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:47.273165 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:47.273175 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:47.273225 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:47.273243 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:47.273388 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:47.273491 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 23:10:47.273646 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260810 23:10:47.275727 4611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:47.276576 4611 log.cc:824] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:47.277040 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:47.278613 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d: No bootstrap required, opened a new log
I20260810 23:10:47.280815 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:47.279238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.154:42655 (local address 127.0.0.1:42541)
0810 23:10:47.279359 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:47.279363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:47.279518 (+ 155us) server_negotiation.cc:407] Connection header received
0810 23:10:47.279595 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:47.279597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:47.279630 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:47.279714 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:47.280436 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:47.280440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:47.280443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:47.280458 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:47.280485 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:47.280497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:47.280591 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:47.280651 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 23:10:47.280687 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:10:47.282500 4611 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } }
I20260810 23:10:47.282714 4611 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:47.282793 4611 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8ae625d640499d902e1abf3e17455d, State: Initialized, Role: FOLLOWER
I20260810 23:10:47.283221 4611 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } }
I20260810 23:10:47.283365 4611 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:47.283452 4611 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:47.283564 4611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:47.284441 4611 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } }
I20260810 23:10:47.284811 4611 leader_election.cc:302] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8ae625d640499d902e1abf3e17455d; no voters:
I20260810 23:10:47.285176 4611 leader_election.cc:288] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:47.285295 4621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:47.285564 4621 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [term 1 LEADER]: Becoming Leader. State: Replica: fc8ae625d640499d902e1abf3e17455d, State: Running, Role: LEADER
I20260810 23:10:47.285940 4621 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } }
I20260810 23:10:47.286513 4611 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:47.287716 4622 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8ae625d640499d902e1abf3e17455d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } } }
I20260810 23:10:47.287858 4622 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:47.288103 4623 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8ae625d640499d902e1abf3e17455d. Latest consensus state: current_term: 1 leader_uuid: "fc8ae625d640499d902e1abf3e17455d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8ae625d640499d902e1abf3e17455d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42541 } } }
I20260810 23:10:47.288197 4623 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:47.288250 4630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:47.291242 4630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:47.296211 4630 catalog_manager.cc:1347] Generated new cluster ID: 353666518cd244ffbc25ec3247e0af3a
I20260810 23:10:47.296280 4630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:47.307324 4630 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:47.307682 4630 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:47.317087 4630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8ae625d640499d902e1abf3e17455d: Generated new TSK 0
I20260810 23:10:47.317830 4630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:47.327544 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4541
2026-08-10T23:10:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:47 dist-test-slave-srgv krb5kdc[4642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:47 dist-test-slave-srgv krb5kdc[4642](info): set up 2 sockets
Aug 10 23:10:47 dist-test-slave-srgv krb5kdc[4642](info): commencing operation
krb5kdc: starting...
W20260810 23:10:48.385972 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:48.512234 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:32827
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:32827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:48.621811 4655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:48.622135 4655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:48.622216 4655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:48.626407 4655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:48.626497 4655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:48.626533 4655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:48.626586 4655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:48.626623 4655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:48.631850 4655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:32827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:48.632965 4655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:48.634043 4655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:48.642604 4663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:48.642645 4655 server_base.cc:1034] running on GCE node
W20260810 23:10:48.642604 4660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:48.642606 4661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:48.643473 4655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:48.644563 4655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:48.645771 4655 hybrid_clock.cc:648] HybridClock initialized: now 1786403448645754 us; error 29 us; skew 500 ppm
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403448, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:48.648419 4655 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:48.649283 4655 webserver.cc:460] Webserver started at http://127.14.75.62:37899/ using document root <none> and password file <none>
I20260810 23:10:48.649899 4655 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:48.649993 4655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:48.650228 4655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:48.652243 4655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "4528d80301514562b1814ab37de5389f"
format_stamp: "Formatted at 2026-08-10 23:10:48 on dist-test-slave-srgv"
server_key: "a71bf22a64dad4366ec6a578b3ff1784"
server_key_iv: "872b5987851016fa022717c7c0276a82"
server_key_version: "encryptionkey@0"
I20260810 23:10:48.652824 4655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "4528d80301514562b1814ab37de5389f"
format_stamp: "Formatted at 2026-08-10 23:10:48 on dist-test-slave-srgv"
server_key: "a71bf22a64dad4366ec6a578b3ff1784"
server_key_iv: "872b5987851016fa022717c7c0276a82"
server_key_version: "encryptionkey@0"
I20260810 23:10:48.656546 4655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:10:48.658942 4671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:48.660115 4655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:48.660256 4655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "4528d80301514562b1814ab37de5389f"
format_stamp: "Formatted at 2026-08-10 23:10:48 on dist-test-slave-srgv"
server_key: "a71bf22a64dad4366ec6a578b3ff1784"
server_key_iv: "872b5987851016fa022717c7c0276a82"
server_key_version: "encryptionkey@0"
I20260810 23:10:48.660367 4655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:48.698881 4655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:48.701822 4655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:48.702049 4655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:48.709267 4655 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:32827
I20260810 23:10:48.709298 4723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:32827 every 8 connection(s)
I20260810 23:10:48.710224 4655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:48.712936 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:48.718253 4724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f: Bootstrap starting.
I20260810 23:10:48.719310 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4655
I20260810 23:10:48.719480 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:48.719820 14636 external_mini_cluster.cc:1428] Setting key 8d31d8004ef0fe1c44ec8f5299d53dae
I20260810 23:10:48.721154 4724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:48.721958 4724 log.cc:824] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:48.724290 4724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f: No bootstrap required, opened a new log
I20260810 23:10:48.728853 4724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } }
I20260810 23:10:48.729117 4724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:48.729213 4724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4528d80301514562b1814ab37de5389f, State: Initialized, Role: FOLLOWER
I20260810 23:10:48.729789 4724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } }
I20260810 23:10:48.729971 4724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:48.730046 4724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:48.730172 4724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:48.731402 4724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } }
I20260810 23:10:48.731755 4724 leader_election.cc:302] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4528d80301514562b1814ab37de5389f; no voters:
I20260810 23:10:48.732079 4724 leader_election.cc:288] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:48.732193 4729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:48.732426 4729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [term 1 LEADER]: Becoming Leader. State: Replica: 4528d80301514562b1814ab37de5389f, State: Running, Role: LEADER
I20260810 23:10:48.732824 4729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } }
I20260810 23:10:48.733532 4724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:48.734576 4730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4528d80301514562b1814ab37de5389f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } } }
I20260810 23:10:48.734714 4730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:48.735076 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4528d80301514562b1814ab37de5389f. Latest consensus state: current_term: 1 leader_uuid: "4528d80301514562b1814ab37de5389f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528d80301514562b1814ab37de5389f" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 32827 } } }
I20260810 23:10:48.735184 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:48.736078 4739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:48.736667 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:48.720925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35517 (local address 127.14.75.62:32827)
0810 23:10:48.721333 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:48.721349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:48.721384 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:10:48.722030 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:48.722047 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:48.722509 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:48.722890 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:48.723853 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:48.724938 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:48.726168 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:48.726653 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:48.726800 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:48.730093 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:48.730100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:48.730119 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:48.730157 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:48.732095 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:48.732800 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:48.732805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:48.732811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:48.732988 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:48.733372 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:48.733375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:48.733377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:48.733926 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:48.734139 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:48.735169 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0810 23:10:48.735410 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260810 23:10:48.739055 4739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:48.739874 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:48.745110 4739 catalog_manager.cc:1347] Generated new cluster ID: 40e341fe4ca146618f0298ecaa2606ad
I20260810 23:10:48.745177 4739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:48.752139 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:48.742137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49001 (local address 127.14.75.62:32827)
0810 23:10:48.742249 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:48.742253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:48.742866 (+ 613us) server_negotiation.cc:407] Connection header received
0810 23:10:48.743017 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:48.743031 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:48.743070 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:48.743157 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:48.743957 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:48.745287 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:48.746438 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:48.746822 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:48.746964 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:48.748548 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:48.748552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:48.748554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:48.748580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:48.750323 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:48.750860 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:48.750863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:48.750865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:48.750950 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:48.751270 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:48.751273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:48.751275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:48.751478 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:48.751644 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:48.751882 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 23:10:48.751983 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:10:48.756667 4739 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:48.757911 4739 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:48.765044 4739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4528d80301514562b1814ab37de5389f: Generated new TSK 0
I20260810 23:10:48.765777 4739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:48.770869 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4655
W20260810 23:10:48.776882 4640 connection.cc:535] client connection to 127.14.75.62:32827 recv error: Network error: failed to read from TLS socket (remote: 127.14.75.62:32827): Connection reset by peer (error 104)
2026-08-10T23:10:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4757](info): set up 2 sockets
Aug 10 23:10:48 dist-test-slave-srgv krb5kdc[4757](info): commencing operation
krb5kdc: starting...
W20260810 23:10:49.830433 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:49 dist-test-slave-srgv krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:49.961388 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45919
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:45919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:50.074807 4771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:50.075173 4771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:50.075263 4771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:50.079433 4771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:50.079545 4771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:50.079586 4771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:50.079624 4771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:50.079658 4771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:50.085680 4771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:45919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:50.086859 4771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:50.087931 4771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:50.096791 4779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:50.096802 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:50.096858 4771 server_base.cc:1034] running on GCE node
W20260810 23:10:50.096807 4777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:50.097697 4771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:50.098896 4771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:50.100097 4771 hybrid_clock.cc:648] HybridClock initialized: now 1786403450100085 us; error 27 us; skew 500 ppm
Aug 10 23:10:50 dist-test-slave-srgv krb5kdc[4757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403450, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:50.103037 4771 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:50.103955 4771 webserver.cc:460] Webserver started at http://127.14.75.62:43797/ using document root <none> and password file <none>
I20260810 23:10:50.104544 4771 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:50.104640 4771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:50.104880 4771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:50.107060 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "a4ae5a84ed664777914c565a36924471"
format_stamp: "Formatted at 2026-08-10 23:10:50 on dist-test-slave-srgv"
server_key: "b6c252c3c9f73bd5bacedcb1bbf27354"
server_key_iv: "99f18bdd23c932a5eb0462c562eecdd0"
server_key_version: "encryptionkey@0"
I20260810 23:10:50.107653 4771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "a4ae5a84ed664777914c565a36924471"
format_stamp: "Formatted at 2026-08-10 23:10:50 on dist-test-slave-srgv"
server_key: "b6c252c3c9f73bd5bacedcb1bbf27354"
server_key_iv: "99f18bdd23c932a5eb0462c562eecdd0"
server_key_version: "encryptionkey@0"
I20260810 23:10:50.111527 4771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:10:50.114238 4786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:50.115484 4771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:10:50.115635 4771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "a4ae5a84ed664777914c565a36924471"
format_stamp: "Formatted at 2026-08-10 23:10:50 on dist-test-slave-srgv"
server_key: "b6c252c3c9f73bd5bacedcb1bbf27354"
server_key_iv: "99f18bdd23c932a5eb0462c562eecdd0"
server_key_version: "encryptionkey@0"
I20260810 23:10:50.115754 4771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:50.134321 4771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:50.137213 4771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:50.137504 4771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:50.144986 4771 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:45919
I20260810 23:10:50.145012 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:45919 every 8 connection(s)
I20260810 23:10:50.146226 4771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:50.149034 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:50.149040 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4771
I20260810 23:10:50.149202 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:50.149602 14636 external_mini_cluster.cc:1428] Setting key 9ce878e9e3dd11ff90e4f69b91d8597e
I20260810 23:10:50.155756 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: Bootstrap starting.
I20260810 23:10:50.158218 4840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:50.158973 4840 log.cc:824] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:50.159340 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:50.151113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39057 (local address 127.14.75.62:45919)
0810 23:10:50.151607 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:50.151623 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:50.151661 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:10:50.152399 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:50.152417 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:50.152772 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:50.153228 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:50.153683 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:50.154714 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:50.155652 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:50.156186 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:50.156380 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:50.157389 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:50.157394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:50.157448 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:50.157857 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:50.157915 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:50.157939 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:50.158083 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:50.158436 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:10:50.158560 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":333,"thread_start_us":151,"threads_started":1}
I20260810 23:10:50.161161 4840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: No bootstrap required, opened a new log
I20260810 23:10:50.162463 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:50.166086 4840 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } }
I20260810 23:10:50.166370 4840 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:50.166417 4840 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ae5a84ed664777914c565a36924471, State: Initialized, Role: FOLLOWER
I20260810 23:10:50.166920 4840 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } }
I20260810 23:10:50.167083 4840 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:50.167146 4840 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:50.167230 4840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:50.168422 4840 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } }
I20260810 23:10:50.168828 4840 leader_election.cc:302] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ae5a84ed664777914c565a36924471; no voters:
I20260810 23:10:50.169215 4840 leader_election.cc:288] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:50.169481 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:50.164859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42813 (local address 127.14.75.62:45919)
0810 23:10:50.164997 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:50.165003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:50.165150 (+ 147us) server_negotiation.cc:407] Connection header received
0810 23:10:50.165234 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:50.165237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:50.165288 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:50.165381 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:50.165687 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:50.166431 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:50.167593 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:50.167842 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:50.167925 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:50.168887 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:50.168890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:50.168891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:50.168903 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:50.168927 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:50.168937 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:50.169056 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:50.169242 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 23:10:50.169276 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:10:50.169299 4850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:50.169770 4850 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [term 1 LEADER]: Becoming Leader. State: Replica: a4ae5a84ed664777914c565a36924471, State: Running, Role: LEADER
I20260810 23:10:50.170109 4850 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } }
I20260810 23:10:50.170646 4840 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:50.171308 4850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ae5a84ed664777914c565a36924471. Latest consensus state: current_term: 1 leader_uuid: "a4ae5a84ed664777914c565a36924471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } } }
I20260810 23:10:50.171449 4850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:50.171828 4851 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ae5a84ed664777914c565a36924471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ae5a84ed664777914c565a36924471" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45919 } } }
I20260810 23:10:50.172019 4851 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471 [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:50.175521 4865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:50.175591 4865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:50.175673 4857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:50.176470 4857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:50.182341 4857 catalog_manager.cc:1347] Generated new cluster ID: 38a9583662df4b5dbfd62b5b295b47dd
I20260810 23:10:50.182427 4857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:50.190407 4857 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:50.191673 4857 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:50.201184 4857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ae5a84ed664777914c565a36924471: Generated new TSK 0
I20260810 23:10:50.202075 4857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:50.208595 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4771
W20260810 23:10:50.214627 4755 connection.cc:535] client connection to 127.14.75.62:45919 recv error: Network error: failed to read from TLS socket (remote: 127.14.75.62:45919): Connection reset by peer (error 104)
2026-08-10T23:10:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:50 dist-test-slave-srgv krb5kdc[4871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:50 dist-test-slave-srgv krb5kdc[4871](info): set up 2 sockets
Aug 10 23:10:50 dist-test-slave-srgv krb5kdc[4871](info): commencing operation
krb5kdc: starting...
W20260810 23:10:51.258018 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:51 dist-test-slave-srgv krb5kdc[4871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:51.382318 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36057
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:51.493162 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:51.493448 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:51.493530 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:51.497505 4884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:51.497563 4884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:51.497582 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:51.497598 4884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:51.497658 4884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:51.502780 4884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:36057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:51.503793 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:51.504802 4884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:51.512848 4893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:51.512862 4890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:51.512877 4889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:51.513192 4884 server_base.cc:1034] running on GCE node
I20260810 23:10:51.513798 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:51.514853 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:51.516041 4884 hybrid_clock.cc:648] HybridClock initialized: now 1786403451516027 us; error 31 us; skew 500 ppm
Aug 10 23:10:51 dist-test-slave-srgv krb5kdc[4871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403451, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:51.518803 4884 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:51.519682 4884 webserver.cc:460] Webserver started at http://127.14.75.62:39883/ using document root <none> and password file <none>
I20260810 23:10:51.520217 4884 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:51.520318 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:51.520542 4884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:51.522708 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "4d1af67f47124eda8220eace36090055"
format_stamp: "Formatted at 2026-08-10 23:10:51 on dist-test-slave-srgv"
server_key: "dfaacf898b339df2b36036cd5084016f"
server_key_iv: "00200f14b9fcc32bb85073e70207ff4c"
server_key_version: "encryptionkey@0"
I20260810 23:10:51.523280 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "4d1af67f47124eda8220eace36090055"
format_stamp: "Formatted at 2026-08-10 23:10:51 on dist-test-slave-srgv"
server_key: "dfaacf898b339df2b36036cd5084016f"
server_key_iv: "00200f14b9fcc32bb85073e70207ff4c"
server_key_version: "encryptionkey@0"
I20260810 23:10:51.527006 4884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:10:51.529313 4900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:51.530520 4884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:10:51.530669 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "4d1af67f47124eda8220eace36090055"
format_stamp: "Formatted at 2026-08-10 23:10:51 on dist-test-slave-srgv"
server_key: "dfaacf898b339df2b36036cd5084016f"
server_key_iv: "00200f14b9fcc32bb85073e70207ff4c"
server_key_version: "encryptionkey@0"
I20260810 23:10:51.530790 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:51.535320 4884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:51.538502 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:51.538702 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:51.545943 4884 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:36057
I20260810 23:10:51.545971 4954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:36057 every 8 connection(s)
I20260810 23:10:51.546877 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:51.548362 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4884
I20260810 23:10:51.548517 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:51.548820 14636 external_mini_cluster.cc:1428] Setting key f580e5a3a119b7d8994a1ce77aae2b45
I20260810 23:10:51.549777 4955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:51.554286 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:51.550182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50287 (local address 127.14.75.62:36057)
0810 23:10:51.550632 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:51.550646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:51.550869 (+ 223us) server_negotiation.cc:407] Connection header received
0810 23:10:51.551606 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:51.551622 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:51.551894 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:51.552163 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:51.553024 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:51.553029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:51.553040 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:51.553051 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:51.553103 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:51.553122 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:51.553229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:51.553372 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:10:51.553564 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":156,"threads_started":1}
I20260810 23:10:51.556113 4955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055: Bootstrap starting.
I20260810 23:10:51.556900 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:51.558390 4955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:51.559180 4955 log.cc:824] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:51.560607 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:51.558968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58075 (local address 127.14.75.62:36057)
0810 23:10:51.559082 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:51.559086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:51.559302 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:10:51.559429 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:51.559431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:51.559460 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:51.559525 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:51.560254 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:51.560258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:51.560261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:51.560273 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:51.560301 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:51.560313 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:51.560398 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:51.560467 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:10:51.560501 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:51.561443 4955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055: No bootstrap required, opened a new log
I20260810 23:10:51.564965 4955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } }
I20260810 23:10:51.565210 4955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:51.565301 4955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1af67f47124eda8220eace36090055, State: Initialized, Role: FOLLOWER
I20260810 23:10:51.565742 4955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } }
I20260810 23:10:51.565925 4955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:51.565992 4955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:51.566089 4955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:51.567065 4955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } }
I20260810 23:10:51.567411 4955 leader_election.cc:302] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d1af67f47124eda8220eace36090055; no voters:
I20260810 23:10:51.567715 4955 leader_election.cc:288] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:51.567844 4965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:51.568081 4965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1af67f47124eda8220eace36090055, State: Running, Role: LEADER
I20260810 23:10:51.568431 4965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } }
I20260810 23:10:51.568923 4955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:51.570094 4967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d1af67f47124eda8220eace36090055. Latest consensus state: current_term: 1 leader_uuid: "4d1af67f47124eda8220eace36090055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } } }
I20260810 23:10:51.570232 4967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:51.570418 4966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d1af67f47124eda8220eace36090055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1af67f47124eda8220eace36090055" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 36057 } } }
I20260810 23:10:51.570565 4966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:51.570622 4974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:51.573736 4974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:51.579336 4974 catalog_manager.cc:1347] Generated new cluster ID: 798c4080a0c54e498c7c1e3a9ab96afb
I20260810 23:10:51.579411 4974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:51.615465 4974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:51.615808 4974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:51.621393 4974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d1af67f47124eda8220eace36090055: Generated new TSK 0
I20260810 23:10:51.622485 4974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:51.637130 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4884
2026-08-10T23:10:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:51 dist-test-slave-srgv krb5kdc[4986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:51 dist-test-slave-srgv krb5kdc[4986](info): set up 2 sockets
Aug 10 23:10:51 dist-test-slave-srgv krb5kdc[4986](info): commencing operation
krb5kdc: starting...
W20260810 23:10:52.698341 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:52 dist-test-slave-srgv krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:52.822682 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42759
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:52.932799 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:52.933123 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:52.933198 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:52.937340 4999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:52.937453 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:52.937502 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:52.937552 4999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:52.937592 4999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:52.944231 4999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:52.945328 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:52.946363 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:52.954782 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:52.954841 4999 server_base.cc:1034] running on GCE node
W20260810 23:10:52.954887 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:52.954825 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:52.955618 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:52.956701 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:52.957880 4999 hybrid_clock.cc:648] HybridClock initialized: now 1786403452957852 us; error 49 us; skew 500 ppm
Aug 10 23:10:52 dist-test-slave-srgv krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403452, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:52.960556 4999 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:52.961537 4999 webserver.cc:460] Webserver started at http://127.14.75.62:45953/ using document root <none> and password file <none>
I20260810 23:10:52.962097 4999 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:52.962159 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:52.962385 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:52.964430 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "710a1878d90546c99242cfb2b67145d2"
format_stamp: "Formatted at 2026-08-10 23:10:52 on dist-test-slave-srgv"
server_key: "7f0ca0efb75b6ecee1b58ef8ebff3b74"
server_key_iv: "88f8290e6a610478f91929f2564f1831"
server_key_version: "encryptionkey@0"
I20260810 23:10:52.965003 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "710a1878d90546c99242cfb2b67145d2"
format_stamp: "Formatted at 2026-08-10 23:10:52 on dist-test-slave-srgv"
server_key: "7f0ca0efb75b6ecee1b58ef8ebff3b74"
server_key_iv: "88f8290e6a610478f91929f2564f1831"
server_key_version: "encryptionkey@0"
I20260810 23:10:52.968787 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:10:52.971180 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:52.972342 4999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:10:52.972476 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "710a1878d90546c99242cfb2b67145d2"
format_stamp: "Formatted at 2026-08-10 23:10:52 on dist-test-slave-srgv"
server_key: "7f0ca0efb75b6ecee1b58ef8ebff3b74"
server_key_iv: "88f8290e6a610478f91929f2564f1831"
server_key_version: "encryptionkey@0"
I20260810 23:10:52.972579 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:52.996577 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:52.999612 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:52.999833 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:53.007122 4999 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42759
I20260810 23:10:53.007149 5069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42759 every 8 connection(s)
I20260810 23:10:53.008123 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:53.009011 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 4999
I20260810 23:10:53.009150 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:53.009496 14636 external_mini_cluster.cc:1428] Setting key 55268ac59d7144e4cb9fa4d2c1d5115e
I20260810 23:10:53.011301 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:53.018136 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2: Bootstrap starting.
Aug 10 23:10:53 dist-test-slave-srgv krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:53.020942 5070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:53.021749 5070 log.cc:824] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:53.024045 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2: No bootstrap required, opened a new log
I20260810 23:10:53.024763 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:53.010939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.14.75.62:42759)
0810 23:10:53.011386 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:53.011407 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:53.011450 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:10:53.012313 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:53.012336 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:53.012755 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:53.013122 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:53.013900 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:53.015399 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:53.016231 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:53.016927 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:53.017114 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:53.019984 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:53.019989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:53.020003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:53.020032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:53.021904 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:53.022405 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:53.022411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:53.022419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:53.022568 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:53.022891 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:53.022893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:53.022894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:53.023272 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:53.023412 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:53.023710 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 23:10:53.024004 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260810 23:10:53.027747 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:53.028257 5070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } }
I20260810 23:10:53.028560 5070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:53.028647 5070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710a1878d90546c99242cfb2b67145d2, State: Initialized, Role: FOLLOWER
I20260810 23:10:53.029122 5070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } }
I20260810 23:10:53.029325 5070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:53.029423 5070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:53.029529 5070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:53.030822 5070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } }
I20260810 23:10:53.031224 5070 leader_election.cc:302] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710a1878d90546c99242cfb2b67145d2; no voters:
I20260810 23:10:53.031590 5070 leader_election.cc:288] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:53.031666 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:53.031885 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [term 1 LEADER]: Becoming Leader. State: Replica: 710a1878d90546c99242cfb2b67145d2, State: Running, Role: LEADER
I20260810 23:10:53.032249 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } }
I20260810 23:10:53.033074 5070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:53.034703 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710a1878d90546c99242cfb2b67145d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } } }
I20260810 23:10:53.034842 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:53.035677 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710a1878d90546c99242cfb2b67145d2. Latest consensus state: current_term: 1 leader_uuid: "710a1878d90546c99242cfb2b67145d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710a1878d90546c99242cfb2b67145d2" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42759 } } }
I20260810 23:10:53.035806 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:53.036119 5091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:53.039378 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:53.030091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41475 (local address 127.14.75.62:42759)
0810 23:10:53.030218 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:53.030225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:53.030367 (+ 142us) server_negotiation.cc:407] Connection header received
0810 23:10:53.030498 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:53.030500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:53.030536 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:53.030598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:53.031503 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:53.032222 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:53.033096 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:53.033442 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:53.033538 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:53.035415 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:53.035418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:53.035422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:53.035447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:53.037122 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:53.037602 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:53.037607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:53.037609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:53.037695 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:53.037967 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:53.037971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:53.037973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:53.038196 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:53.038332 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:53.039081 (+ 749us) server_negotiation.cc:299] Negotiation successful
0810 23:10:53.039205 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:53.039780 5091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:53.045516 5091 catalog_manager.cc:1347] Generated new cluster ID: e3feab39aaaa49f5abd2dd13f95d9f1d
I20260810 23:10:53.045598 5091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:53.057276 5091 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:53.058344 5091 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:53.064460 5091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710a1878d90546c99242cfb2b67145d2: Generated new TSK 0
I20260810 23:10:53.065117 5091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:53.071702 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 4999
2026-08-10T23:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:53 dist-test-slave-srgv krb5kdc[5103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:53 dist-test-slave-srgv krb5kdc[5103](info): set up 2 sockets
Aug 10 23:10:53 dist-test-slave-srgv krb5kdc[5103](info): commencing operation
krb5kdc: starting...
W20260810 23:10:54.126303 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:54 dist-test-slave-srgv krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:54.250562 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42133
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:39841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:42133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:54.361083 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:54.361343 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:54.361409 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:54.365545 5116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:54.365607 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:54.365628 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:54.365685 5116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:54.365717 5116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:54.370913 5116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:39841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:42133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:54.371938 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:54.372933 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:54.381145 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:54.381263 5116 server_base.cc:1034] running on GCE node
W20260810 23:10:54.381139 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:54.381139 5124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:54.382064 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:54.383103 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:54.384271 5116 hybrid_clock.cc:648] HybridClock initialized: now 1786403454384257 us; error 34 us; skew 500 ppm
Aug 10 23:10:54 dist-test-slave-srgv krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403454, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:54.387036 5116 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:54.387957 5116 webserver.cc:460] Webserver started at http://127.14.75.62:44313/ using document root <none> and password file <none>
I20260810 23:10:54.388505 5116 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:54.388567 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:54.388803 5116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:54.390858 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "3134e31551504c4f8e64a506e2fe41f8"
format_stamp: "Formatted at 2026-08-10 23:10:54 on dist-test-slave-srgv"
server_key: "3c862e21a1691bebce886b4a49e73dfa"
server_key_iv: "48902fd82970c9cf8799636df7d3718c"
server_key_version: "encryptionkey@0"
I20260810 23:10:54.391414 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "3134e31551504c4f8e64a506e2fe41f8"
format_stamp: "Formatted at 2026-08-10 23:10:54 on dist-test-slave-srgv"
server_key: "3c862e21a1691bebce886b4a49e73dfa"
server_key_iv: "48902fd82970c9cf8799636df7d3718c"
server_key_version: "encryptionkey@0"
I20260810 23:10:54.395066 5116 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 23:10:54.397369 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:54.398571 5116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:10:54.398695 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "3134e31551504c4f8e64a506e2fe41f8"
format_stamp: "Formatted at 2026-08-10 23:10:54 on dist-test-slave-srgv"
server_key: "3c862e21a1691bebce886b4a49e73dfa"
server_key_iv: "48902fd82970c9cf8799636df7d3718c"
server_key_version: "encryptionkey@0"
I20260810 23:10:54.398797 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:54.428028 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:54.431058 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:54.431289 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:54.438912 5116 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:42133
I20260810 23:10:54.438938 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:42133 every 8 connection(s)
I20260810 23:10:54.439982 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:54.442790 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:54.447247 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5116
I20260810 23:10:54.447407 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:54.447685 14636 external_mini_cluster.cc:1428] Setting key 16ac040b8b4331c1e4a2416063cd17d0
I20260810 23:10:54.448962 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8: Bootstrap starting.
I20260810 23:10:54.451261 5186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:54.452019 5186 log.cc:824] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:54.453931 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8: No bootstrap required, opened a new log
I20260810 23:10:54.456501 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:54.449182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58615 (local address 127.14.75.62:42133)
0810 23:10:54.449652 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:54.449668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:54.449703 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:10:54.450333 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:54.450350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:54.450679 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:54.451002 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:54.451468 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:54.452473 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:54.453354 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:54.453867 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:54.454029 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:54.454898 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:54.454903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:54.454915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:54.455315 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:54.455378 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:54.455399 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:54.455504 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:54.455641 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:10:54.455835 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":168,"threads_started":1}
I20260810 23:10:54.457712 5186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } }
I20260810 23:10:54.457975 5186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:54.458041 5186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3134e31551504c4f8e64a506e2fe41f8, State: Initialized, Role: FOLLOWER
I20260810 23:10:54.458519 5186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } }
I20260810 23:10:54.458649 5186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:54.458806 5186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:54.458925 5186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:54.459429 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:54.460005 5186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } }
I20260810 23:10:54.460345 5186 leader_election.cc:302] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3134e31551504c4f8e64a506e2fe41f8; no voters:
I20260810 23:10:54.460662 5186 leader_election.cc:288] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:54.460731 5192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:54.460953 5192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3134e31551504c4f8e64a506e2fe41f8, State: Running, Role: LEADER
I20260810 23:10:54.461299 5192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } }
I20260810 23:10:54.462153 5186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:54.463042 5196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3134e31551504c4f8e64a506e2fe41f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } } }
I20260810 23:10:54.463172 5196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:54.463563 5197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3134e31551504c4f8e64a506e2fe41f8. Latest consensus state: current_term: 1 leader_uuid: "3134e31551504c4f8e64a506e2fe41f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3134e31551504c4f8e64a506e2fe41f8" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 42133 } } }
I20260810 23:10:54.463725 5197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:54.464670 5205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:54.466436 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:54.461927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43691 (local address 127.14.75.62:42133)
0810 23:10:54.462037 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:54.462041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:54.462052 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:10:54.462085 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:54.462093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:54.462128 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:54.462209 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:54.462647 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:54.463536 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:54.464687 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:54.465035 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:54.465140 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:54.466010 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:54.466014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:54.466016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:54.466029 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:54.466061 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:54.466072 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:54.466149 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:54.466240 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 23:10:54.466300 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260810 23:10:54.467983 5205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:54.473578 5205 catalog_manager.cc:1347] Generated new cluster ID: 768a2c201582492c9d75117bcf61225f
I20260810 23:10:54.473659 5205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:54.483742 5205 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:54.484699 5205 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:54.489385 5205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3134e31551504c4f8e64a506e2fe41f8: Generated new TSK 0
I20260810 23:10:54.490062 5205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:54.499003 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5116
2026-08-10T23:10:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:54 dist-test-slave-srgv krb5kdc[5217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:54 dist-test-slave-srgv krb5kdc[5217](info): set up 2 sockets
Aug 10 23:10:54 dist-test-slave-srgv krb5kdc[5217](info): commencing operation
krb5kdc: starting...
W20260810 23:10:55.562434 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:55 dist-test-slave-srgv krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:55.687824 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45713
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:45713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:55.797709 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:55.798017 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:55.798059 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:55.802193 5230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:55.802264 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:55.802284 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:55.802347 5230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:55.802376 5230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:55.807859 5230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:45713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:45713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:55.808961 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:55.809994 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:55.818122 5239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:55.818173 5237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:55.818397 5230 server_base.cc:1034] running on GCE node
W20260810 23:10:55.818122 5236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:55.819070 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:55.820122 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:55.821300 5230 hybrid_clock.cc:648] HybridClock initialized: now 1786403455821279 us; error 45 us; skew 500 ppm
Aug 10 23:10:55 dist-test-slave-srgv krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403455, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:55.824113 5230 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:55.824991 5230 webserver.cc:460] Webserver started at http://127.14.75.62:45399/ using document root <none> and password file <none>
I20260810 23:10:55.825580 5230 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:55.825650 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:55.825829 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:55.827872 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "e466878322aa4b1a8830ff8909bee249"
format_stamp: "Formatted at 2026-08-10 23:10:55 on dist-test-slave-srgv"
server_key: "2d48ef6f15edab53aa776c77b620366d"
server_key_iv: "2fa3557586b652f24f8ef338aba6e4f7"
server_key_version: "encryptionkey@0"
I20260810 23:10:55.828408 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "e466878322aa4b1a8830ff8909bee249"
format_stamp: "Formatted at 2026-08-10 23:10:55 on dist-test-slave-srgv"
server_key: "2d48ef6f15edab53aa776c77b620366d"
server_key_iv: "2fa3557586b652f24f8ef338aba6e4f7"
server_key_version: "encryptionkey@0"
I20260810 23:10:55.832173 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:10:55.834575 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:55.835790 5230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:10:55.835899 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "e466878322aa4b1a8830ff8909bee249"
format_stamp: "Formatted at 2026-08-10 23:10:55 on dist-test-slave-srgv"
server_key: "2d48ef6f15edab53aa776c77b620366d"
server_key_iv: "2fa3557586b652f24f8ef338aba6e4f7"
server_key_version: "encryptionkey@0"
I20260810 23:10:55.835987 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:55.840493 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:55.843495 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:55.843644 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:55.850438 5230 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:45713
I20260810 23:10:55.850497 5298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:45713 every 8 connection(s)
I20260810 23:10:55.851377 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:55.854089 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:55.854429 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5230
I20260810 23:10:55.854559 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:55.854825 14636 external_mini_cluster.cc:1428] Setting key 0762c5453fc78179805d465d9c0a1c47
I20260810 23:10:55.860152 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:55.856079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45445 (local address 127.14.75.62:45713)
0810 23:10:55.856488 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:55.856507 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:55.856649 (+ 142us) server_negotiation.cc:407] Connection header received
0810 23:10:55.857410 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:55.857455 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:55.857807 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:55.858171 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:55.858826 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:55.858832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:55.858845 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:55.858856 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:55.858914 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:55.858940 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:55.859077 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:55.859147 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:10:55.859306 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260810 23:10:55.861152 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249: Bootstrap starting.
I20260810 23:10:55.862848 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:55.863489 5299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:55.864243 5299 log.cc:824] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:55.866496 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:55.864908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40745 (local address 127.14.75.62:45713)
0810 23:10:55.865035 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:55.865039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:55.865226 (+ 187us) server_negotiation.cc:407] Connection header received
0810 23:10:55.865345 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:55.865348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:55.865382 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:55.865518 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:55.866158 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:55.866161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:55.866164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:10:55.866176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:55.866206 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:10:55.866220 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:55.866320 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:55.866352 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 23:10:55.866387 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:10:55.866551 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249: No bootstrap required, opened a new log
I20260810 23:10:55.870213 5299 raft_consensus.cc:348] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } }
I20260810 23:10:55.870443 5299 raft_consensus.cc:374] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:10:55.870487 5299 raft_consensus.cc:729] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e466878322aa4b1a8830ff8909bee249, State: Initialized, Role: FOLLOWER
I20260810 23:10:55.870846 5299 consensus_queue.cc:260] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } }
I20260810 23:10:55.870958 5299 raft_consensus.cc:388] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:10:55.871002 5299 raft_consensus.cc:482] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:10:55.871073 5299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:55.872025 5299 raft_consensus.cc:504] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } }
I20260810 23:10:55.872331 5299 leader_election.cc:302] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e466878322aa4b1a8830ff8909bee249; no voters:
I20260810 23:10:55.872592 5299 leader_election.cc:288] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:10:55.872710 5310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:55.872932 5310 raft_consensus.cc:686] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [term 1 LEADER]: Becoming Leader. State: Replica: e466878322aa4b1a8830ff8909bee249, State: Running, Role: LEADER
I20260810 23:10:55.873291 5310 consensus_queue.cc:237] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } }
I20260810 23:10:55.873869 5299 sys_catalog.cc:564] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:55.875121 5312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e466878322aa4b1a8830ff8909bee249. Latest consensus state: current_term: 1 leader_uuid: "e466878322aa4b1a8830ff8909bee249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } } }
I20260810 23:10:55.875121 5311 sys_catalog.cc:455] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e466878322aa4b1a8830ff8909bee249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e466878322aa4b1a8830ff8909bee249" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 45713 } } }
I20260810 23:10:55.875265 5311 sys_catalog.cc:458] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:55.875264 5312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249 [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:55.875756 5319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:55.878499 5319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:55.883834 5319 catalog_manager.cc:1347] Generated new cluster ID: 2b29dfb592854ded85498ce8964b6c2b
I20260810 23:10:55.883904 5319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:55.939154 5319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:55.939533 5319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:55.952700 5319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e466878322aa4b1a8830ff8909bee249: Generated new TSK 0
I20260810 23:10:55.953688 5319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:10:55.971637 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5230
2026-08-10T23:10:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1472 ms)
[----------] 18 tests from AuthTokenIssuingTest (25962 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:56 dist-test-slave-srgv krb5kdc[5331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:56 dist-test-slave-srgv krb5kdc[5331](info): set up 2 sockets
Aug 10 23:10:56 dist-test-slave-srgv krb5kdc[5331](info): commencing operation
krb5kdc: starting...
W20260810 23:10:57.026064 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:10:57.150731 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38663
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:10:57.263190 5344 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:10:57.263490 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:57.263515 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:57.263554 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:57.267711 5344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:57.267776 5344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:57.267795 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:57.267812 5344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:57.267827 5344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:57.273262 5344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:38663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:57.274385 5344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:57.275414 5344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:57.283968 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:57.284065 5344 server_base.cc:1034] running on GCE node
W20260810 23:10:57.283986 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:57.283991 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:57.284829 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:57.285933 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:57.287091 5344 hybrid_clock.cc:648] HybridClock initialized: now 1786403457287069 us; error 40 us; skew 500 ppm
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:57.290042 5344 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:10:57.291013 5344 webserver.cc:460] Webserver started at http://127.14.75.62:38545/ using document root <none> and password file <none>
I20260810 23:10:57.291577 5344 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:57.291639 5344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:57.291885 5344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:57.294026 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "b866c2039c42405aa8d4be21d971f7ef"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "691890d3bdf618ae299187e76bd69a29"
server_key_iv: "9cd8a71a2dee378804d9b23d99d6b66d"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.294605 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "b866c2039c42405aa8d4be21d971f7ef"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "691890d3bdf618ae299187e76bd69a29"
server_key_iv: "9cd8a71a2dee378804d9b23d99d6b66d"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.298341 5344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:10:57.300648 5359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:57.301904 5344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:10:57.302040 5344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "b866c2039c42405aa8d4be21d971f7ef"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "691890d3bdf618ae299187e76bd69a29"
server_key_iv: "9cd8a71a2dee378804d9b23d99d6b66d"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.302145 5344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:57.316725 5344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:57.319604 5344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:57.319815 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:57.327056 5344 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:38663
I20260810 23:10:57.327369 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:38663 every 8 connection(s)
I20260810 23:10:57.328146 5344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:10:57.330978 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:57.334729 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.337730 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5344
I20260810 23:10:57.337877 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:10:57.338140 14636 external_mini_cluster.cc:1428] Setting key 4332baf997dc328403bbadcd41fcb003
WARNING: no policy specified for kudu/127.14.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.61@KRBTEST.COM" created.
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:57.358328 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38663 (local address 127.0.0.1:53194)
0810 23:10:57.337398 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.337473 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.338453 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.339243 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.339247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.339595 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.339862 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.339875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.341025 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.341029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.341830 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.341840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.342520 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.342528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.342694 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.346513 (+ 3819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.346553 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.350881 (+ 4328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.354934 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.354943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.354953 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.355192 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.355631 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.355633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.355634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.355747 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.356639 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.356645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.356928 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.357143 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.357511 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":160,"threads_started":1}
I20260810 23:10:57.358332 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.14.75.62:38663)
0810 23:10:57.337441 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.337462 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.337594 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:10:57.338770 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.338778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.339019 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.339224 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.340020 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.340889 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.341949 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.342312 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.342499 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.351093 (+ 8594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.351099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.351111 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.351137 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.354770 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.355340 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.355349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.355354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.355515 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.355856 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.355861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.355866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.356520 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.356685 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.357352 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.357646 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":390,"spinlock_wait_cycles":27392,"thread_start_us":167,"threads_started":1}
I20260810 23:10:57.360852 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.361667 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.361373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54930)
0810 23:10:57.361531 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.361604 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 23:10:57.361912 5360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
I20260810 23:10:57.362856 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.362628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54934)
0810 23:10:57.362759 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.362803 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:10:57.363008 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34955: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
I20260810 23:10:57.368011 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:34955
--webserver_interface=127.14.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:10:57.412367 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } attempt: 1
I20260810 23:10:57.413374 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.412973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54938)
0810 23:10:57.413223 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.413296 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 23:10:57.414141 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.413929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54952)
0810 23:10:57.414035 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.414087 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:10:57.414294 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34955: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
W20260810 23:10:57.481703 5421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:10:57.481981 5421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:57.482051 5421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:57.482115 5421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:10:57.482447 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } attempt: 2
I20260810 23:10:57.483213 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.482932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54958)
0810 23:10:57.483069 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.483138 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:10:57.483952 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.483760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54968)
0810 23:10:57.483855 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.483900 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 23:10:57.484098 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:34955: Network error: Client connection negotiation failed: client connection to 127.14.75.61:34955: connect: Connection refused (error 111)
W20260810 23:10:57.486586 5421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:57.486644 5421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:57.486685 5421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:57.486722 5421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:57.486779 5421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:57.492695 5421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:57.493842 5421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:57.494879 5421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:57.503351 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:57.503355 5429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:57.503522 5421 server_base.cc:1034] running on GCE node
W20260810 23:10:57.503414 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:57.504245 5421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:57.505357 5421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:57.506564 5421 hybrid_clock.cc:648] HybridClock initialized: now 1786403457506549 us; error 28 us; skew 500 ppm
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:57.509295 5421 init.cc:377] Logged in from keytab as kudu/127.14.75.61@KRBTEST.COM (short username kudu)
I20260810 23:10:57.510267 5421 webserver.cc:460] Webserver started at http://127.14.75.61:38035/ using document root <none> and password file <none>
I20260810 23:10:57.510844 5421 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:57.510937 5421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:57.511170 5421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:57.513231 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data/instance:
uuid: "e42f114abc5b4181a330131b9017edbc"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "5e14f5240c38017ff6b9e0f74b3b6a71"
server_key_iv: "7376d46a508e46742727aacb7a5ba1c3"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.513873 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal/instance:
uuid: "e42f114abc5b4181a330131b9017edbc"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "5e14f5240c38017ff6b9e0f74b3b6a71"
server_key_iv: "7376d46a508e46742727aacb7a5ba1c3"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.517588 5421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:10:57.519915 5436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:57.521102 5421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:57.521243 5421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "e42f114abc5b4181a330131b9017edbc"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "5e14f5240c38017ff6b9e0f74b3b6a71"
server_key_iv: "7376d46a508e46742727aacb7a5ba1c3"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.521361 5421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:57.560627 5421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:57.563735 5421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:57.563961 5421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:57.571568 5421 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.61:34955
I20260810 23:10:57.571853 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.61:34955 every 8 connection(s)
I20260810 23:10:57.572624 5421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
I20260810 23:10:57.575173 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5421
I20260810 23:10:57.575325 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-1/wal/instance
I20260810 23:10:57.575260 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } attempt: 3
I20260810 23:10:57.575510 5491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:57.575682 14636 external_mini_cluster.cc:1428] Setting key 743edf0e26122b55dc93cadd6111405b
I20260810 23:10:57.580518 5491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.75.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.60@KRBTEST.COM" created.
I20260810 23:10:57.590107 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.575794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54970)
0810 23:10:57.576015 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.576032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.576136 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.578632 (+ 2496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.578635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.578860 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.578997 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.579006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.580988 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.580992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.581847 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.581859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.583091 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.583097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.583304 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.583933 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.583949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.585571 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.587725 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.587727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.587729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.588033 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.588460 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.588466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.588470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.588565 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.589391 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.589397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.589540 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.589666 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.589836 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:57.591192 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.576524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.14.75.61:34955)
0810 23:10:57.577040 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.577062 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.577110 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:10:57.577994 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.578013 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.578350 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.578687 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.579122 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.580781 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.581995 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.582846 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.583636 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.585746 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.585757 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.585781 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.585815 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.587605 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.588156 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.588163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.588171 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.588356 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.588716 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.588719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.588721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.589262 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.589442 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.589830 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.590154 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":165,"threads_started":1}
I20260810 23:10:57.593567 5414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.594393 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.594096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46402)
0810 23:10:57.594235 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.594313 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:10:57.595187 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.594896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46404)
0810 23:10:57.595067 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.595127 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 23:10:57.595333 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.595759 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.582396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38663 (local address 127.0.0.1:53198)
0810 23:10:57.582772 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.582820 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.583843 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.584949 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.584958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.585498 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.585687 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.585696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.586904 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.586906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.587515 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.587524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.588675 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.588680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.588810 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.589375 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.589389 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.591184 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.593893 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.593902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.593907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.594224 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.594580 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.594584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.594587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.594684 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.595160 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.595167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.595298 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.595430 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.595600 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260810 23:10:57.596174 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.582423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.14.75.62:38663)
0810 23:10:57.582549 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.582553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.582893 (+ 340us) server_negotiation.cc:407] Connection header received
0810 23:10:57.584694 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.584700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.584765 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.584856 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.585813 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.586585 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.587636 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.588554 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.588671 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.591370 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.591374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.591376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.591401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.593762 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.594353 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.594356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.594358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.594438 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.594822 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.594825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.594827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.595033 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.595921 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.595995 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.596086 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:10:57.597021 5491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:10:57.605978 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.597559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54982)
0810 23:10:57.597658 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.597671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.597742 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.597989 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.597991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.598120 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.598204 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.598209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.599178 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.599180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.599669 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.599675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.600085 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.600087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.600156 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.600563 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.600573 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.602700 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.604429 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.604432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.604433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.604623 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.604918 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.604920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.604922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.604997 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.605393 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.605395 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.605542 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.605631 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.605804 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 23:10:57.605980 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.597596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.14.75.61:34955)
0810 23:10:57.597749 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.597753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.597807 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:10:57.597846 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.597851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.597899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.597979 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.598317 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.599086 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.599779 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.599995 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.600069 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.602815 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.602817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.602819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.602840 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.604329 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.604704 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.604707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.604708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.604831 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.605072 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.605074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.605075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.605235 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.605485 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.605686 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.605865 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":69}
I20260810 23:10:57.606752 5491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
I20260810 23:10:57.607429 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.607167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46412)
0810 23:10:57.607275 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.607367 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:10:57.608093 5437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.608397 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:36725
--webserver_interface=127.14.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:10:57.609007 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.608820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46428)
0810 23:10:57.608920 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.608962 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:10:57.609202 5491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.626513 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 1
I20260810 23:10:57.627342 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.627056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46438)
0810 23:10:57.627202 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.627269 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:10:57.628052 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.627853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46448)
0810 23:10:57.627953 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.627997 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:10:57.628206 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.658491 5491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 1
I20260810 23:10:57.659375 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.659087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46462)
0810 23:10:57.659232 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.659301 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:10:57.660094 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.659887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46466)
0810 23:10:57.659987 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.660038 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:10:57.660254 5491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.703379 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 2
I20260810 23:10:57.704207 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.703898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46472)
0810 23:10:57.704044 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.704132 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:10:57.704934 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.704732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46478)
0810 23:10:57.704834 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.704885 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:10:57.705080 5414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
W20260810 23:10:57.721518 5499 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:10:57.721807 5499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:10:57.721864 5499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:10:57.721951 5499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:10:57.726203 5499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:10:57.726267 5499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:10:57.726286 5499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:10:57.726333 5499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:10:57.726366 5499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:10:57.728411 5491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 2
I20260810 23:10:57.729197 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.728904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46490)
0810 23:10:57.729051 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.729125 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:10:57.729956 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.729774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46500)
0810 23:10:57.729865 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.729909 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:10:57.730103 5491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:36725: Network error: Client connection negotiation failed: client connection to 127.14.75.60:36725: connect: Connection refused (error 111)
I20260810 23:10:57.732102 5499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:38663,127.14.75.61:34955,127.14.75.60:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:10:57.733191 5499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:10:57.734323 5499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:10:57.742588 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:57.742588 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:10:57.742614 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:10:57.742947 5499 server_base.cc:1034] running on GCE node
I20260810 23:10:57.743501 5499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:10:57.744496 5499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:10:57.745683 5499 hybrid_clock.cc:648] HybridClock initialized: now 1786403457745657 us; error 51 us; skew 500 ppm
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:10:57.748534 5499 init.cc:377] Logged in from keytab as kudu/127.14.75.60@KRBTEST.COM (short username kudu)
I20260810 23:10:57.749480 5499 webserver.cc:460] Webserver started at http://127.14.75.60:37997/ using document root <none> and password file <none>
I20260810 23:10:57.750044 5499 fs_manager.cc:359] Metadata directory not provided
I20260810 23:10:57.750135 5499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:10:57.750371 5499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:10:57.752401 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data/instance:
uuid: "5bc4c5499d6148c7a07c14756d926962"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "58b6e8bea6f5a6e5e7f3179e5f0ce15b"
server_key_iv: "2003069fce94e0e0278c7d9ec00a1670"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.752974 5499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal/instance:
uuid: "5bc4c5499d6148c7a07c14756d926962"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "58b6e8bea6f5a6e5e7f3179e5f0ce15b"
server_key_iv: "2003069fce94e0e0278c7d9ec00a1670"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.756801 5499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:10:57.759136 5515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:10:57.760347 5499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:10:57.760491 5499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "5bc4c5499d6148c7a07c14756d926962"
format_stamp: "Formatted at 2026-08-10 23:10:57 on dist-test-slave-srgv"
server_key: "58b6e8bea6f5a6e5e7f3179e5f0ce15b"
server_key_iv: "2003069fce94e0e0278c7d9ec00a1670"
server_key_version: "encryptionkey@0"
I20260810 23:10:57.760603 5499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:10:57.777099 5499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:10:57.780558 5499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:10:57.780771 5499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:10:57.787708 5499 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.60:36725
I20260810 23:10:57.788007 5567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.60:36725 every 8 connection(s)
I20260810 23:10:57.788738 5499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
I20260810 23:10:57.791577 5568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:10:57.795267 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5499
I20260810 23:10:57.795415 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786403171263156-14636-0/minicluster-data/master-2/wal/instance
I20260810 23:10:57.795742 14636 external_mini_cluster.cc:1428] Setting key 729cc2948cdf8ccfcdd93db47526cb71
I20260810 23:10:57.795567 5568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.804237 5414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 3
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:10:57.809808 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.797520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.14.75.62:38663)
0810 23:10:57.797667 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.797671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.797988 (+ 317us) server_negotiation.cc:407] Connection header received
0810 23:10:57.798578 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.798581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.798630 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.798741 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.799827 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.800613 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.801930 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.802241 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.802346 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.805571 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.805576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.805582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.805622 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.807308 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.807882 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.807886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.807887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.807987 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.808387 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.808390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.808392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.808635 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.808714 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.809301 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.809468 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:10:57.810659 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.797451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:38663 (local address 127.0.0.1:53212)
0810 23:10:57.797844 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.797893 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.798378 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.798803 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.798808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.799286 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.799678 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.799699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.800723 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.800726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.801818 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.801829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.802391 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.802394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.802532 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.803129 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.803157 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.805429 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.807411 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.807421 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.807431 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.807779 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.808069 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.808075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.808080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.808261 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.808719 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.808729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.809042 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.810004 (+ 962us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.810188 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":162,"threads_started":1}
I20260810 23:10:57.810918 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.797136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.14.75.60:36725)
0810 23:10:57.797578 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.797598 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.797638 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:10:57.798367 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.798384 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.798608 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.798784 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.799736 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.800907 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.802064 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.802564 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.802642 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.805805 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.805813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.805830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.805859 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.807905 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.808357 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.808366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.808370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.808460 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.808776 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.808780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.808781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.809022 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.809260 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.809510 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.809734 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:10:57.811913 5568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.815475 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.804692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46526)
0810 23:10:57.804820 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.804834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.804917 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.805644 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.805647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.805830 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.805986 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.805994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.807140 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.807143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.807986 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.807998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.808813 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.808817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.808895 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.809758 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.809771 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.811614 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.813871 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.813874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.813876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.814151 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.814471 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.814474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.814476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.814577 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.814961 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.814964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.815063 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.815166 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.815318 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:10:57.815500 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.804993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.14.75.60:36725)
0810 23:10:57.805344 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.805351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.805362 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:10:57.805388 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.805390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.805505 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.805599 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.806126 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.806975 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.808120 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.808468 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.808571 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.811763 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.811768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.811772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.811797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.813736 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.814270 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.814273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.814275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.814376 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.814700 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.814703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.814704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.814862 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.814967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.815224 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.815334 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:10:57.819108 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: Bootstrap starting.
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:10:57.821271 5491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } attempt: 3
I20260810 23:10:57.822084 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.812891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.14.75.61:34955)
0810 23:10:57.813050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.813058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.813072 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:10:57.813108 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.813110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.813153 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.813256 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.813798 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.814457 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.815983 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.816277 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.816395 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.818340 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.818343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.818346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.818372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.819778 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.820193 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.820199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.820203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.820305 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.820642 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.820646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.820648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.820898 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.821011 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.821792 (+ 781us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.821918 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:10:57.822570 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.812406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:34955 (local address 127.0.0.1:54984)
0810 23:10:57.812542 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.812552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.812648 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.813287 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.813291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.813546 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.813655 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.813663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.814546 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.814549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.815500 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.815509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.816392 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.816395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.816483 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.816903 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.816913 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.818212 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.819888 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.819894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.819896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.820087 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.820473 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.820475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.820476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.820548 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.821465 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.821468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.821574 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.822268 (+ 694us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.822416 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 23:10:57.823099 5414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:57.823630 5568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:10:57.824129 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.813631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.14.75.62:38663)
0810 23:10:57.813749 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.813755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.813766 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:10:57.813796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.813799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.813835 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.813908 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.814623 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.815353 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.816446 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.817041 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.817146 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.819933 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.819936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.819938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.819962 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.821742 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.822243 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.822247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.822248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.822327 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.822617 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.822620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.822622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.822836 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.823012 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.823536 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.823641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:10:57.825778 5414 log.cc:824] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:10:57.828117 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: No bootstrap required, opened a new log
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:10:57.831743 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46536)
0810 23:10:57.822020 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.822034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.822105 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.822651 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.822655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.822810 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.822883 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.822888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.824087 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.824090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.824799 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.824808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.825273 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.825277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.825361 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.825938 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.825954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.827459 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.829960 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.829964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.829966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.830225 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.830553 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.830556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.830558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.830651 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.831130 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.831139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.831252 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.831353 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.831505 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
Aug 10 23:10:57 dist-test-slave-srgv krb5kdc[5331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:10:57.832469 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.822336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.14.75.60:36725)
0810 23:10:57.822437 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.822441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.822452 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:10:57.822483 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.822486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.822522 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.823127 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.823231 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.823936 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.824933 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.825155 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.825262 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.827601 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.827604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.827607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.827628 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.829837 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.830351 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.830355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.830358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.830445 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.830774 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.830778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.830780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.831010 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.832058 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.832133 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.832286 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:10:57.832906 5414 raft_consensus.cc:348] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.833168 5414 raft_consensus.cc:729] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b866c2039c42405aa8d4be21d971f7ef, State: Initialized, Role: FOLLOWER
I20260810 23:10:57.834129 5414 consensus_queue.cc:260] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.834648 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.824094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:36725 (local address 127.0.0.1:46542)
0810 23:10:57.824306 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 23:10:57.824320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:10:57.824389 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:10:57.825676 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:10:57.825679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:10:57.825871 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:10:57.825964 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.825971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.826964 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.826967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.827763 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:10:57.827772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.828514 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.828520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:10:57.828625 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.829061 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:10:57.829081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:10:57.830657 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:10:57.832882 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.832886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.832889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.833133 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.833725 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:10:57.833727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:10:57.833729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:10:57.833811 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:10:57.834204 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:10:57.834207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:10:57.834280 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 23:10:57.834373 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:10:57.834514 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 23:10:57.835135 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.824318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.14.75.60:36725)
0810 23:10:57.825389 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.825394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.825405 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:10:57.825459 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.825463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.825508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.825585 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.826089 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.826844 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.827890 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.828291 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.828399 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.830785 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.830788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.830791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.830812 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.832533 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.833526 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.833531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.833536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.833624 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.833907 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.833909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.833911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.834082 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.834163 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.834911 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.835014 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014}
I20260810 23:10:57.835793 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: Bootstrap starting.
I20260810 23:10:57.836016 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.825984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.14.75.61:34955)
0810 23:10:57.826078 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.826082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.826374 (+ 292us) server_negotiation.cc:407] Connection header received
0810 23:10:57.826414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.826420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.826467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.826534 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.827342 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.828002 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.829039 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.829343 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.829465 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.832410 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.832414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.832416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.832440 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.834084 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.834589 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.834595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.834600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.834688 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.835009 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.835012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.835014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.835218 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.835399 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.835761 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.835877 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:10:57.836668 5414 sys_catalog.cc:564] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:57.837813 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: Bootstrap starting.
I20260810 23:10:57.838550 5491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:57.839416 5491 log.cc:824] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:57.840247 5568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: Neither blocks nor log segments found. Creating new log.
I20260810 23:10:57.840888 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.841012 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.841065 5568 log.cc:824] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: Log is configured to *not* fsync() on all Append() calls
I20260810 23:10:57.843003 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:10:57.843218 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: No bootstrap required, opened a new log
I20260810 23:10:57.843218 5568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: No bootstrap required, opened a new log
I20260810 23:10:57.842110 5344 master_runner.cc:186] Error getting master registration for 127.14.75.61:34955: OK, instance_id { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" instance_seqno: 1786403457569220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:10:57.845793 5344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:10:57.845974 5387 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b866c2039c42405aa8d4be21d971f7ef"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53224
I20260810 23:10:57.846176 5387 raft_consensus.cc:482] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:10:57.846323 5387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:57.847335 5387 raft_consensus.cc:504] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.847307 5491 raft_consensus.cc:348] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.847308 5568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.847525 5491 raft_consensus.cc:729] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42f114abc5b4181a330131b9017edbc, State: Initialized, Role: FOLLOWER
I20260810 23:10:57.847525 5568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc4c5499d6148c7a07c14756d926962, State: Initialized, Role: FOLLOWER
I20260810 23:10:57.847870 5568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.847872 5491 consensus_queue.cc:260] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.848338 5387 leader_election.cc:288] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [CANDIDATE]: Term 1 election: Requested vote from peers e42f114abc5b4181a330131b9017edbc (127.14.75.61:34955), 5bc4c5499d6148c7a07c14756d926962 (127.14.75.60:36725)
I20260810 23:10:57.848378 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.848645 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.848975 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.849195 5491 sys_catalog.cc:564] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:57.849500 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b866c2039c42405aa8d4be21d971f7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e42f114abc5b4181a330131b9017edbc"
I20260810 23:10:57.849706 5464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:57.849797 5542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b866c2039c42405aa8d4be21d971f7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5bc4c5499d6148c7a07c14756d926962"
I20260810 23:10:57.851065 5464 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b866c2039c42405aa8d4be21d971f7ef in term 1.
I20260810 23:10:57.851171 5542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:10:57.851490 5360 leader_election.cc:302] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b866c2039c42405aa8d4be21d971f7ef, e42f114abc5b4181a330131b9017edbc; no voters:
I20260810 23:10:57.851768 5576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:10:57.851964 5576 raft_consensus.cc:686] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [term 1 LEADER]: Becoming Leader. State: Replica: b866c2039c42405aa8d4be21d971f7ef, State: Running, Role: LEADER
I20260810 23:10:57.852284 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } }
I20260810 23:10:57.852499 5542 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b866c2039c42405aa8d4be21d971f7ef in term 1.
I20260810 23:10:57.849808 5568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:10:57.854596 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b866c2039c42405aa8d4be21d971f7ef. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.854712 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: This master's current role is: LEADER
W20260810 23:10:57.857165 5608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:57.857259 5608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:57.862205 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.851875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.14.75.61:34955)
0810 23:10:57.852116 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.852121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.852365 (+ 244us) server_negotiation.cc:407] Connection header received
0810 23:10:57.852550 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.852553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.852600 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.852652 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.853768 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.854500 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.856032 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.856363 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.856439 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.858395 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.858399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.858402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.858428 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.860156 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.860723 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.860727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.860729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.860818 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.861153 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.861158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.861159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.861389 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.861519 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.861901 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.862026 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260810 23:10:57.865291 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.851996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.14.75.60:36725)
0810 23:10:57.852627 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.852634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.855869 (+ 3235us) server_negotiation.cc:407] Connection header received
0810 23:10:57.855991 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.856006 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.856062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.856155 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.857124 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.857832 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.859041 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.859339 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.859411 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.861187 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.861191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.861193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.861215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.863289 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.863828 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.863832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.863835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.863921 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.864263 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.864266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.864268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.864513 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.864643 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.865037 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.865154 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552}
I20260810 23:10:57.849172 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:10:57.842097 5587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:57.865789 5587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:57.865886 5624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:10:57.866928 5624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:10:57.872304 5464 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [term 1 FOLLOWER]: Refusing update from remote peer b866c2039c42405aa8d4be21d971f7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:10:57.872412 5542 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [term 1 FOLLOWER]: Refusing update from remote peer b866c2039c42405aa8d4be21d971f7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 23:10:57.872820 5623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:10:57.872910 5623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:10:57.873086 5576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:57.873296 5610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:10:57.873517 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:57.855646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.14.75.62:38663)
0810 23:10:57.855768 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.855772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.863693 (+ 7921us) server_negotiation.cc:407] Connection header received
0810 23:10:57.863864 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.863867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.863915 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.863964 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.864787 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.865667 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.867631 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:57.867951 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:57.868025 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:57.869847 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:10:57.869850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:10:57.869853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:10:57.869878 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:10:57.871610 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.872108 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.872112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.872114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.872211 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:10:57.872523 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:10:57.872526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:10:57.872528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:10:57.872737 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:10:57.872839 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:57.873227 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:10:57.873344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:10:57.877008 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b866c2039c42405aa8d4be21d971f7ef. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.877144 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.877506 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b866c2039c42405aa8d4be21d971f7ef. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.877624 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.881702 5576 sys_catalog.cc:455] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.881831 5576 sys_catalog.cc:458] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:57.882007 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.882110 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.882282 5628 mvcc.cc:205] Tried to move back new op lower bound from 7317108563438141440 to 7317108563367776256. Current Snapshot: MvccSnapshot[applied={T|T < 7317108563438141440}]
I20260810 23:10:57.882342 5627 mvcc.cc:205] Tried to move back new op lower bound from 7317108563438141440 to 7317108563367776256. Current Snapshot: MvccSnapshot[applied={T|T < 7317108563438141440}]
I20260810 23:10:57.885764 5624 catalog_manager.cc:1347] Generated new cluster ID: e07d2af447fa41c6bf5d51a18ba3abf4
I20260810 23:10:57.885831 5624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:10:57.888703 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.888823 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef [sys.catalog]: This master's current role is: LEADER
I20260810 23:10:57.890004 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b866c2039c42405aa8d4be21d971f7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b866c2039c42405aa8d4be21d971f7ef" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 38663 } } peers { permanent_uuid: "e42f114abc5b4181a330131b9017edbc" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 34955 } } peers { permanent_uuid: "5bc4c5499d6148c7a07c14756d926962" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 36725 } } }
I20260810 23:10:57.890127 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:10:57.890587 5629 mvcc.cc:205] Tried to move back new op lower bound from 7317108563438141440 to 7317108563367776256. Current Snapshot: MvccSnapshot[applied={T|T < 7317108563438141440}]
I20260810 23:10:57.903115 5624 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:10:57.904383 5624 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:10:57.921512 5624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b866c2039c42405aa8d4be21d971f7ef: Generated new TSK 0
I20260810 23:10:57.922408 5624 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:10:57.962604 5417 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.958814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.14.75.62:38663)
0810 23:10:57.958942 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.958946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.960542 (+ 1596us) server_negotiation.cc:407] Connection header received
0810 23:10:57.961044 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.961048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.961169 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.961270 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.962374 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53242: BlockingRecv error: recv got EOF from 127.0.0.1:53242 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 23:10:57.964221 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.958306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.14.75.61:34955)
0810 23:10:57.958439 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.958443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.963145 (+ 4702us) server_negotiation.cc:407] Connection header received
0810 23:10:57.963336 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.963340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.963392 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.963444 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.964087 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55008: BlockingRecv error: recv got EOF from 127.0.0.1:55008 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:10:57.965756 5574 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.959596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.14.75.60:36725)
0810 23:10:57.963007 (+ 3411us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.963012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.964888 (+ 1876us) server_negotiation.cc:407] Connection header received
0810 23:10:57.964996 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.964998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.965055 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.965141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.965636 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46562: BlockingRecv error: recv got EOF from 127.0.0.1:46562 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 23:10:57.969782 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.968162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.14.75.61:34955)
0810 23:10:57.968266 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.968270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.968778 (+ 508us) server_negotiation.cc:407] Connection header received
0810 23:10:57.968988 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.968991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.969067 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.969128 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.969673 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55020: BlockingRecv error: recv got EOF from 127.0.0.1:55020 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 23:10:57.974825 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.973393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.14.75.61:34955)
0810 23:10:57.973533 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.973537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.973988 (+ 451us) server_negotiation.cc:407] Connection header received
0810 23:10:57.974176 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.974179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.974226 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.974274 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.974732 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55032: BlockingRecv error: recv got EOF from 127.0.0.1:55032 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:10:57.980211 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.978872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.14.75.61:34955)
0810 23:10:57.978960 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.978964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.979402 (+ 438us) server_negotiation.cc:407] Connection header received
0810 23:10:57.979568 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.979571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.979614 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.979667 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.980102 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55044: BlockingRecv error: recv got EOF from 127.0.0.1:55044 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 23:10:57.987926 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.986376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.14.75.61:34955)
0810 23:10:57.986514 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.986518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.987006 (+ 488us) server_negotiation.cc:407] Connection header received
0810 23:10:57.987209 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.987212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.987259 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.987309 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.987819 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55046: BlockingRecv error: recv got EOF from 127.0.0.1:55046 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:10:58.000064 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:57.998385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.14.75.61:34955)
0810 23:10:57.998502 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:57.998506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:57.999066 (+ 560us) server_negotiation.cc:407] Connection header received
0810 23:10:57.999298 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:57.999301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:57.999356 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:57.999408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:57.999948 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55052: BlockingRecv error: recv got EOF from 127.0.0.1:55052 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 23:10:58.021137 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:58.018934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.14.75.61:34955)
0810 23:10:58.019070 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:58.019075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:58.019939 (+ 864us) server_negotiation.cc:407] Connection header received
0810 23:10:58.020217 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:58.020221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:58.020288 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:58.020350 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:58.020979 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55064: BlockingRecv error: recv got EOF from 127.0.0.1:55064 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:10:58.057611 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:58.055878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.14.75.61:34955)
0810 23:10:58.056000 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:58.056005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:58.056572 (+ 567us) server_negotiation.cc:407] Connection header received
0810 23:10:58.056778 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:58.056782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:58.056834 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:58.056885 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:58.057477 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55076: BlockingRecv error: recv got EOF from 127.0.0.1:55076 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:10:58.125470 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:58.124146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.14.75.61:34955)
0810 23:10:58.124300 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:58.124305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:58.124535 (+ 230us) server_negotiation.cc:407] Connection header received
0810 23:10:58.124655 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:58.124658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:58.124712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:58.124799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:58.125329 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55090: BlockingRecv error: recv got EOF from 127.0.0.1:55090 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:10:58.256606 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:58.255445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.14.75.61:34955)
0810 23:10:58.255635 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:58.255639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:58.255723 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:10:58.255869 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:58.255871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:58.255922 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:58.255998 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:58.256451 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55094: BlockingRecv error: recv got EOF from 127.0.0.1:55094 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 23:10:58.515952 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:10:58.514770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.14.75.61:34955)
0810 23:10:58.514920 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:58.514926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:58.515027 (+ 101us) server_negotiation.cc:407] Connection header received
0810 23:10:58.515204 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:58.515207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:58.515270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:58.515359 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:10:58.515836 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55098: BlockingRecv error: recv got EOF from 127.0.0.1:55098 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:10:58.858515 5608 catalog_manager.cc:1259] Loaded cluster ID: e07d2af447fa41c6bf5d51a18ba3abf4
I20260810 23:10:58.858623 5608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: loading cluster ID for follower catalog manager: success
I20260810 23:10:58.860334 5608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: acquiring CA information for follower catalog manager: success
I20260810 23:10:58.861588 5608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e42f114abc5b4181a330131b9017edbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:10:58.874203 5623 catalog_manager.cc:1259] Loaded cluster ID: e07d2af447fa41c6bf5d51a18ba3abf4
I20260810 23:10:58.874322 5623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: loading cluster ID for follower catalog manager: success
I20260810 23:10:58.876021 5623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: acquiring CA information for follower catalog manager: success
I20260810 23:10:58.877224 5623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bc4c5499d6148c7a07c14756d926962: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:10:59.033777 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:59.030137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.14.75.61:34955)
0810 23:10:59.030381 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:59.030385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:59.030402 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:10:59.030536 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:59.030539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:59.030665 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:59.030777 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:59.030991 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.031507 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.032307 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.032516 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.032589 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:59.032864 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:59.033285 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:59.033376 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:59.033497 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 23:10:59.033576 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260810 23:10:59.040458 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:59.036893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.14.75.60:36725)
0810 23:10:59.037114 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:59.037118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:59.037131 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:10:59.037211 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:59.037213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:59.037401 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:59.037543 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:59.037794 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.038307 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.039173 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.039398 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.039478 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:59.039725 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:59.040140 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:59.040233 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:59.040288 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 23:10:59.040336 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260810 23:10:59.047540 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:59.044008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.14.75.60:36725)
0810 23:10:59.044142 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:59.044148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:59.044160 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:10:59.044201 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:59.044206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:59.044258 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:59.044328 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:59.044586 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.045077 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.046300 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.046638 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.046737 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:59.046981 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:59.047088 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:59.047155 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:59.047326 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:10:59.047379 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:10:59.047605 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:59.043418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.14.75.61:34955)
0810 23:10:59.043648 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:59.043652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:59.043888 (+ 236us) server_negotiation.cc:407] Connection header received
0810 23:10:59.044099 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:59.044103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:59.044160 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:59.044258 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:59.044497 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.045238 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.046420 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.046685 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.046777 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:59.047100 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:59.047185 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:59.047269 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:59.047418 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 23:10:59.047478 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260810 23:10:59.048269 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:10:59.043393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.14.75.62:38663)
0810 23:10:59.043655 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:10:59.043660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:10:59.043675 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:10:59.043866 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:10:59.043872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:10:59.044537 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:10:59.044653 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:10:59.045014 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.045707 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.046675 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:10:59.047018 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:10:59.047117 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:10:59.047514 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:10:59.047878 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:10:59.047967 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:10:59.048055 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:10:59.048112 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260810 23:10:59.051319 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5344
I20260810 23:10:59.059202 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5421
I20260810 23:10:59.066258 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5499
2026-08-10T23:10:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3096 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:10:59 dist-test-slave-srgv krb5kdc[5720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:10:59 dist-test-slave-srgv krb5kdc[5720](info): set up 2 sockets
Aug 10 23:10:59 dist-test-slave-srgv krb5kdc[5720](info): commencing operation
krb5kdc: starting...
W20260810 23:11:00.133214 14636 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/test-user.keytab.
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.75.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
Entry for principal kudu/127.14.75.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab.
I20260810 23:11:00.258713 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41041
--webserver_interface=127.14.75.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:11:00.374908 5733 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:11:00.375216 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:11:00.375241 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:11:00.375361 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:11:00.379945 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:11:00.380034 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:11:00.380055 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:11:00.380080 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:11:00.380168 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:11:00.385773 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.62.keytab
--principal=kudu/127.14.75.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.62:41041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.75.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:11:00.386909 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:11:00.387995 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:11:00.396656 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:11:00.396713 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:11:00.396656 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:11:00.397133 5733 server_base.cc:1034] running on GCE node
I20260810 23:11:00.397787 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:11:00.398969 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:11:00.400168 5733 hybrid_clock.cc:648] HybridClock initialized: now 1786403460400146 us; error 35 us; skew 500 ppm
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:11:00.403100 5733 init.cc:377] Logged in from keytab as kudu/127.14.75.62@KRBTEST.COM (short username kudu)
I20260810 23:11:00.404013 5733 webserver.cc:460] Webserver started at http://127.14.75.62:34801/ using document root <none> and password file <none>
I20260810 23:11:00.404608 5733 fs_manager.cc:359] Metadata directory not provided
I20260810 23:11:00.404707 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:11:00.404953 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:11:00.407133 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data/instance:
uuid: "40ad38a0c249473eb28de452d7fb3333"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "18f5e878f4d678144f8b5480f56e41ad"
server_key_iv: "3aea70704e59f86ba5e246eb20d806dd"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.407749 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance:
uuid: "40ad38a0c249473eb28de452d7fb3333"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "18f5e878f4d678144f8b5480f56e41ad"
server_key_iv: "3aea70704e59f86ba5e246eb20d806dd"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.411648 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:11:00.414103 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:11:00.415288 5733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:11:00.415436 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal
uuid: "40ad38a0c249473eb28de452d7fb3333"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "18f5e878f4d678144f8b5480f56e41ad"
server_key_iv: "3aea70704e59f86ba5e246eb20d806dd"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.415555 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:11:00.430367 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:11:00.433857 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:11:00.434074 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:11:00.440997 5733 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.62:41041
I20260810 23:11:00.441320 5802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.62:41041 every 8 connection(s)
I20260810 23:11:00.442049 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/data/info.pb
I20260810 23:11:00.444720 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:11:00.445468 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5733
I20260810 23:11:00.445633 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-0/wal/instance
I20260810 23:11:00.445967 14636 external_mini_cluster.cc:1428] Setting key 32dfc252defc523e65a17eaadf446b87
I20260810 23:11:00.449069 5803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.75.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.61@KRBTEST.COM" created.
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:11:00.465332 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.451797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41041 (local address 127.0.0.1:36122)
0810 23:11:00.452289 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.452341 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.453281 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.454175 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.454179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.454517 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.454880 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.454894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.456261 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.456267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.456984 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.456993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.457664 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.457667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.457811 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.458475 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.458491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.460085 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.462403 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.462412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.462419 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.462649 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.463023 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.463025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.463026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.463138 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.463950 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.463955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.464089 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.464232 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.464484 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":42,"thread_start_us":145,"threads_started":1}
I20260810 23:11:00.465339 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.451798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.14.75.62:41041)
0810 23:11:00.452368 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.452383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.452432 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:11:00.453664 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.453671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.453946 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.454147 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.455034 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.456130 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.457095 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.457516 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.457635 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.460239 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.460244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.460256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.460277 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.462293 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.462756 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.462766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.462769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.462921 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.463238 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.463244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.463247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.463800 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.463961 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.464355 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.464570 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":433,"thread_start_us":138,"threads_started":1}
I20260810 23:11:00.467803 5803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:11:00.468570 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.468311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60442)
0810 23:11:00.468437 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.468507 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:11:00.468779 5750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
I20260810 23:11:00.469730 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.469522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60452)
0810 23:11:00.469628 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.469678 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:11:00.469877 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:35889: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
Entry for principal kudu/127.14.75.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab.
I20260810 23:11:00.474462 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:35889
--webserver_interface=127.14.75.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:11:00.519177 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } attempt: 1
I20260810 23:11:00.520093 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.519751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60460)
0810 23:11:00.519926 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.519995 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 23:11:00.520912 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.520705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60464)
0810 23:11:00.520816 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.520861 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:11:00.521134 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:35889: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
I20260810 23:11:00.589304 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } attempt: 2
W20260810 23:11:00.589223 5810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:11:00.589641 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:11:00.589710 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:11:00.589805 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:11:00.590236 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.589946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60476)
0810 23:11:00.590097 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.590160 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:11:00.590998 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.590801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60482)
0810 23:11:00.590900 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.590944 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:11:00.591140 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.61:35889: Network error: Client connection negotiation failed: client connection to 127.14.75.61:35889: connect: Connection refused (error 111)
W20260810 23:11:00.594359 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:11:00.594411 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:11:00.594457 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:11:00.594498 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:11:00.594550 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:11:00.599915 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.61.keytab
--principal=kudu/127.14.75.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.61:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.75.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:11:00.600991 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:11:00.602005 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:11:00.610301 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:11:00.610380 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:11:00.610432 5810 server_base.cc:1034] running on GCE node
W20260810 23:11:00.610306 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:11:00.611127 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:11:00.612205 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:11:00.613368 5810 hybrid_clock.cc:648] HybridClock initialized: now 1786403460613364 us; error 35 us; skew 500 ppm
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:11:00.616151 5810 init.cc:377] Logged in from keytab as kudu/127.14.75.61@KRBTEST.COM (short username kudu)
I20260810 23:11:00.617076 5810 webserver.cc:460] Webserver started at http://127.14.75.61:37247/ using document root <none> and password file <none>
I20260810 23:11:00.617694 5810 fs_manager.cc:359] Metadata directory not provided
I20260810 23:11:00.617758 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:11:00.618010 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:11:00.620232 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data/instance:
uuid: "8ac3d57c00114183b7f36f4f7694bb31"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "3783497a78037a97909c560202e5f78c"
server_key_iv: "70bdcfed199d767d4c633cd4d7cada03"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.620877 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal/instance:
uuid: "8ac3d57c00114183b7f36f4f7694bb31"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "3783497a78037a97909c560202e5f78c"
server_key_iv: "70bdcfed199d767d4c633cd4d7cada03"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.624545 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:11:00.627039 5825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:11:00.628263 5810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:11:00.628403 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal
uuid: "8ac3d57c00114183b7f36f4f7694bb31"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "3783497a78037a97909c560202e5f78c"
server_key_iv: "70bdcfed199d767d4c633cd4d7cada03"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.628508 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:11:00.643415 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:11:00.646246 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:11:00.646445 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:11:00.653609 5810 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.61:35889
I20260810 23:11:00.653923 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.61:35889 every 8 connection(s)
I20260810 23:11:00.654599 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/data/info.pb
I20260810 23:11:00.657359 5878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:11:00.661000 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:11:00.661624 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5810
I20260810 23:11:00.661758 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-1/wal/instance
I20260810 23:11:00.662077 14636 external_mini_cluster.cc:1428] Setting key 1da96350522950bdbab67c2828cfdda6
WARNING: no policy specified for kudu/127.14.75.60@KRBTEST.COM; defaulting to no policy
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.75.60@KRBTEST.COM" created.
I20260810 23:11:00.675112 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.663194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.14.75.62:41041)
0810 23:11:00.663375 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.663382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.664187 (+ 805us) server_negotiation.cc:407] Connection header received
0810 23:11:00.665103 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.665110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.665177 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.665285 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.666464 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.667002 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.668147 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.668387 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.668499 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.671287 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.671290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.671292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.671318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.672679 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.673315 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.673319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.673321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.673439 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.673839 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.673845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.673847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.674098 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.674211 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.674708 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.674860 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:11:00.675995 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.663503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41041 (local address 127.0.0.1:36138)
0810 23:11:00.663981 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.664066 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.664896 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.665478 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.665484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.665831 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.666304 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.666317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.667139 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.667148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.668012 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.668020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.668539 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.668544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.668715 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.669352 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.669367 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.671128 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.672793 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.672798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.672809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.673201 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.673539 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.673545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.673550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.673729 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.674207 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.674212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.674457 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.674616 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.674878 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
I20260810 23:11:00.677623 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:11:00.682320 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } attempt: 3
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:11:00.689451 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.678173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60494)
0810 23:11:00.678323 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.678342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.678429 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.678950 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.678956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.679164 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.679302 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.679313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.680272 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.680277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.681211 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.681224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.681889 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.681896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.682023 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.682579 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.682595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.685377 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.687640 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.687643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.687644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.687842 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.688207 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.688210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.688211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.688332 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.688936 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.688942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.689051 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.689164 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.689270 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:11:00.689728 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.678294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.14.75.61:35889)
0810 23:11:00.678608 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.678617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.678634 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:11:00.678668 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.678672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.678809 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.678912 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.679425 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.680147 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.681333 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.681760 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.681842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.685542 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.685551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.685562 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.685582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.687512 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.688016 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.688023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.688024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.688112 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.688433 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.688436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.688438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.688801 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.688956 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.689324 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.689544 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"spinlock_wait_cycles":1792,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
Entry for principal kudu/127.14.75.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab.
I20260810 23:11:00.691983 5878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:11:00.692246 14636 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
/tmp/dist-test-taskmgqrPD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:46705
--webserver_interface=127.14.75.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.75.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:11:00.692713 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.692460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40666)
0810 23:11:00.692574 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.692655 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:11:00.692895 5829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.692961 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.682768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60508)
0810 23:11:00.682928 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.682943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.683029 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.683368 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.683371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.683526 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.683625 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.683630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.684415 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.684418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.685740 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.685750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.686311 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.686346 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.686487 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.686908 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.686920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.688255 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.691085 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.691090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.691092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.691318 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.691606 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.691609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.691613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.691707 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.692249 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.692252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.692370 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.692469 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.692654 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:11:00.693696 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.682803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.14.75.61:35889)
0810 23:11:00.683074 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.683080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.683091 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:11:00.683119 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.683121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.683152 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.683257 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.683725 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.684321 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.685895 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.686179 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.686298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.688430 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.688434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.688437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.688468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.690975 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.691421 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.691425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.691427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.691512 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.691813 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.691817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.691819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.692113 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.693168 (+ 1055us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.693255 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.693389 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":2304,"thread_start_us":123,"threads_started":1}
I20260810 23:11:00.693971 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.693751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40680)
0810 23:11:00.693871 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.693910 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 23:11:00.694128 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.694422 5803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:11:00.695164 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.694920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40696)
0810 23:11:00.695032 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.695081 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:11:00.695874 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.695646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40706)
0810 23:11:00.695788 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.695828 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 23:11:00.696000 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.727183 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 1
I20260810 23:11:00.728087 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.727789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40716)
0810 23:11:00.727946 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.728006 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:11:00.728809 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.728593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40718)
0810 23:11:00.728707 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.728757 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:11:00.728940 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.743392 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 1
I20260810 23:11:00.744253 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.743950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40726)
0810 23:11:00.744120 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.744180 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:11:00.744958 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.744745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40742)
0810 23:11:00.744852 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.744903 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:11:00.745080 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.804122 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 2
I20260810 23:11:00.804986 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.804688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40748)
0810 23:11:00.804843 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.804908 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:11:00.805748 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.805539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40750)
0810 23:11:00.805644 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.805689 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:11:00.805903 5803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
W20260810 23:11:00.809120 5886 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:11:00.809394 5886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:11:00.809485 5886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:11:00.809535 5886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:11:00.813251 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 2
W20260810 23:11:00.813643 5886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:11:00.813719 5886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:11:00.813784 5886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:11:00.813858 5886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:11:00.813915 5886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:11:00.814014 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.813718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40752)
0810 23:11:00.813860 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.813920 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:11:00.814703 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.814492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40754)
0810 23:11:00.814594 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.814648 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:11:00.814823 5878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.75.60:46705: Network error: Client connection negotiation failed: client connection to 127.14.75.60:46705: connect: Connection refused (error 111)
I20260810 23:11:00.819589 5886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.75.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.75.62:41041,127.14.75.61:35889,127.14.75.60:46705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/krb5kdc/kudu_127.14.75.60.keytab
--principal=kudu/127.14.75.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.75.60:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.75.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:11:00.820724 5886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:11:00.821820 5886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:11:00.830200 5894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:11:00.830232 5892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:11:00.830204 5891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:11:00.830760 5886 server_base.cc:1034] running on GCE node
I20260810 23:11:00.831377 5886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:11:00.832566 5886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:11:00.833726 5886 hybrid_clock.cc:648] HybridClock initialized: now 1786403460833699 us; error 39 us; skew 500 ppm
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:11:00.836550 5886 init.cc:377] Logged in from keytab as kudu/127.14.75.60@KRBTEST.COM (short username kudu)
I20260810 23:11:00.837585 5886 webserver.cc:460] Webserver started at http://127.14.75.60:33485/ using document root <none> and password file <none>
I20260810 23:11:00.838161 5886 fs_manager.cc:359] Metadata directory not provided
I20260810 23:11:00.838227 5886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:11:00.838495 5886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:11:00.840690 5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data/instance:
uuid: "249a6816b34044b3a83ead885db896a2"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "e31a1a80d80ec4acf79dd40863644540"
server_key_iv: "243008ecf89b12efc9809db6a6097525"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.841288 5886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal/instance:
uuid: "249a6816b34044b3a83ead885db896a2"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "e31a1a80d80ec4acf79dd40863644540"
server_key_iv: "243008ecf89b12efc9809db6a6097525"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.845165 5886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:11:00.847749 5901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:11:00.849066 5886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:11:00.849273 5886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data,/tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal
uuid: "249a6816b34044b3a83ead885db896a2"
format_stamp: "Formatted at 2026-08-10 23:11:00 on dist-test-slave-srgv"
server_key: "e31a1a80d80ec4acf79dd40863644540"
server_key_iv: "243008ecf89b12efc9809db6a6097525"
server_key_version: "encryptionkey@0"
I20260810 23:11:00.849460 5886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:11:00.865491 5886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:11:00.868434 5886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:11:00.868650 5886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:11:00.875790 5886 rpc_server.cc:307] RPC server started. Bound to: 127.14.75.60:46705
I20260810 23:11:00.876106 5955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.75.60:46705 every 8 connection(s)
I20260810 23:11:00.876768 5886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/data/info.pb
I20260810 23:11:00.879592 5956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:11:00.879946 14636 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu as pid 5886
I20260810 23:11:00.880113 14636 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmgqrPD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786403171263156-14636-0/minicluster-data/master-2/wal/instance
I20260810 23:11:00.880417 14636 external_mini_cluster.cc:1428] Setting key c93030aaf224ee86ddb7fe22494e6f6a
I20260810 23:11:00.884388 5956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:11:00.895061 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.881785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.14.75.60:46705)
0810 23:11:00.882199 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.882214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.882248 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:11:00.882873 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.882889 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.883215 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.883478 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.884415 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.885484 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.886529 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.887111 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.887234 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.889897 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.889904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.889914 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.889946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.891758 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.892251 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.892256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.892260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.892374 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.892662 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.892666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.892668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.893207 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.893461 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.893756 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.894053 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260810 23:11:00.895893 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.886166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.62:41041 (local address 127.0.0.1:36142)
0810 23:11:00.886602 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.886640 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.886773 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.887075 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.887081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.887568 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.887739 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.887750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.888478 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.888481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.889247 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.889260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.889897 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.889901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.890021 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.890628 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.890646 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.892345 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.894330 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.894338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.894341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.894548 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.894823 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.894825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.894826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.894894 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.895408 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.895413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.895509 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.895632 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.895765 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":171,"threads_started":1}
I20260810 23:11:00.895961 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.886225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.14.75.62:41041)
0810 23:11:00.886385 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.886389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.886716 (+ 327us) server_negotiation.cc:407] Connection header received
0810 23:11:00.886884 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.886886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.886939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.887020 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.887869 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.888352 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.889378 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.889759 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.889861 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.892481 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.892484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.892487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.892513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.894210 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.894643 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.894646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.894648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.894734 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.894978 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.894981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.894983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.895271 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.895378 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.895704 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.895824 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:11:00.897217 5956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.62@KRBTEST.COM
I20260810 23:11:00.905113 5803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 3
I20260810 23:11:00.906069 5878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } attempt: 3
I20260810 23:11:00.907264 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.897724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.61:35889 (local address 127.0.0.1:60516)
0810 23:11:00.897962 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.897975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.898041 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.898377 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.898380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.898551 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.898687 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.898695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.899670 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.899672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.900523 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.900533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.901132 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.901138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.901237 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.901837 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.901856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.903353 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.905469 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.905476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.905481 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.905746 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.906110 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.906113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.906114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.906204 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.906700 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.906703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.906808 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.906924 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.907078 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260810 23:11:00.907282 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.897992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.14.75.61:35889)
0810 23:11:00.898137 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.898141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.898155 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:11:00.898201 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.898207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.898264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.898339 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.898802 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.899566 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.900640 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.901004 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.901119 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.903497 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.903500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.903502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.903525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.905298 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.905869 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.905874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.905877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.905973 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.906317 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.906323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.906324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.906546 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.906622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.906990 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.907134 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:11:00.908228 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.897706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.14.75.62:41041)
0810 23:11:00.897852 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.897855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.897929 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:11:00.898080 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.898083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.898130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.898216 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.898971 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.899655 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.900728 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.901041 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.901161 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.904173 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.904179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.904182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.904212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.905966 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.906498 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.906502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.906503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.906584 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.906904 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.906910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.906915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.907150 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.907395 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.907762 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.907892 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:11:00.908526 5956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.62@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.61@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.75.60@KRBTEST.COM for kudu/127.14.75.60@KRBTEST.COM
I20260810 23:11:00.915783 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.905713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.14.75.60:46705)
0810 23:11:00.905851 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.905858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.905916 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:11:00.906073 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.906077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.906127 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.906984 (+ 857us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.907073 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.907750 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.908845 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.909164 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.909627 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.911625 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.911629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.911631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.911654 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.913304 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.914468 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.914474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.914479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.914561 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.914894 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.914897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.914899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.915111 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.915219 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.915531 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.915634 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:11:00.915824 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.905635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40768)
0810 23:11:00.905859 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.905874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.905973 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.906262 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.906266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.906437 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.906547 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.906555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.907881 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.907884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.908554 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.908562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.909290 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.909294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.909388 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.909891 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.909910 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.911481 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.914099 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.914102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.914104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.914364 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.914675 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.914680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.914684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.914778 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.915217 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.915223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.915338 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.915459 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.915608 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
Aug 10 23:11:00 dist-test-slave-srgv krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.75.61@KRBTEST.COM
I20260810 23:11:00.920392 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.906456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40780)
0810 23:11:00.906688 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.906701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.906763 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.909875 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.909880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.910072 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.910178 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.910185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.911405 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.911415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.912232 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.912242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.912934 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.912937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.913019 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.913469 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.913484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.915951 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.918665 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.918671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.918674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.918997 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.919310 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.919315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.919396 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.919921 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.919923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.920020 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.920123 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.920243 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"spinlock_wait_cycles":2432}
I20260810 23:11:00.920388 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.909014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.75.60:46705 (local address 127.0.0.1:40796)
0810 23:11:00.909158 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:11:00.909172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:11:00.909244 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:11:00.909977 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:11:00.909982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:11:00.910153 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:11:00.910234 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.910240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.911773 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.911777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.912638 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:11:00.912652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.913297 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.913300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:11:00.913395 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.913865 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:11:00.913878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:11:00.915567 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:11:00.918816 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.918819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.918821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.919041 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.919315 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:11:00.919320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:11:00.919396 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:11:00.919909 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:11:00.919912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:11:00.919987 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 23:11:00.920083 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:11:00.920229 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":4}
I20260810 23:11:00.920454 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: Bootstrap starting.
I20260810 23:11:00.920938 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.909249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.14.75.60:46705)
0810 23:11:00.909738 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.909747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.909758 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:11:00.909786 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.909788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.909853 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.910336 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.910924 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.911661 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.912778 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.913176 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.913622 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.916233 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.916237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.916239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.916262 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.918710 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919153 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.919158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.919162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.919226 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919497 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.919500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.919502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.919807 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.920561 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.920648 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.920770 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":425,"spinlock_wait_cycles":4736,"thread_start_us":106,"threads_started":1}
I20260810 23:11:00.921381 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.910229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.14.75.61:35889)
0810 23:11:00.910329 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.910332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.910490 (+ 158us) server_negotiation.cc:407] Connection header received
0810 23:11:00.910687 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.910690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.910729 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.910773 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.911585 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.912793 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.914332 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.914640 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.914714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.917568 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.917571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.917573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.917596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.919355 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919904 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.919907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.919908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.920000 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.920330 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.920333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.920335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.920538 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.920728 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.921096 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.921210 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":896}
I20260810 23:11:00.921406 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.906762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.14.75.60:46705)
0810 23:11:00.909625 (+ 2863us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.909635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.909649 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:11:00.909680 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.909684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.909729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.910289 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.910425 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.911269 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.912387 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.912809 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.913246 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.916091 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.916094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.916096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.916117 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.918505 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919114 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.919117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.919119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.919200 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.919498 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.919500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.919502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.919807 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.921056 (+ 1249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.921135 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.921250 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":2774,"spinlock_wait_cycles":55680,"thread_start_us":112,"threads_started":1}
I20260810 23:11:00.923949 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: Bootstrap starting.
I20260810 23:11:00.924001 5956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: Bootstrap starting.
I20260810 23:11:00.925165 14636 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:11:00.926538 5956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: Neither blocks nor log segments found. Creating new log.
I20260810 23:11:00.926566 5878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: Neither blocks nor log segments found. Creating new log.
I20260810 23:11:00.927187 5956 log.cc:824] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:11:00.927336 5878 log.cc:824] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: Log is configured to *not* fsync() on all Append() calls
I20260810 23:11:00.926445 5775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "40ad38a0c249473eb28de452d7fb3333"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36154
I20260810 23:11:00.929131 5878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: No bootstrap required, opened a new log
I20260810 23:11:00.929243 5956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: No bootstrap required, opened a new log
I20260810 23:11:00.930343 5803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: Neither blocks nor log segments found. Creating new log.
I20260810 23:11:00.931182 5803 log.cc:824] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: Log is configured to *not* fsync() on all Append() calls
I20260810 23:11:00.933405 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: No bootstrap required, opened a new log
I20260810 23:11:00.933579 5878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.933816 5878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac3d57c00114183b7f36f4f7694bb31, State: Initialized, Role: FOLLOWER
I20260810 23:11:00.934082 5956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.934173 5878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.934309 5956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249a6816b34044b3a83ead885db896a2, State: Initialized, Role: FOLLOWER
I20260810 23:11:00.934629 5956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.935173 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.935472 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:11:00.935513 5878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:11:00.935854 5956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:11:00.936682 5803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.936904 5803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ad38a0c249473eb28de452d7fb3333, State: Initialized, Role: FOLLOWER
I20260810 23:11:00.937196 5803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.938531 5803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:11:00.938736 5775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:11:00.938856 5775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:11:00.940253 5775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.940685 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.940791 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:11:00.941470 5991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:11:00.941545 5991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:11:00.939904 5886 master_runner.cc:186] Error getting master registration for 127.14.75.62:41041: OK, instance_id { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" instance_seqno: 1786403460439105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:11:00.943471 5886 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:11:00.944386 5775 leader_election.cc:288] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac3d57c00114183b7f36f4f7694bb31 (127.14.75.61:35889), 249a6816b34044b3a83ead885db896a2 (127.14.75.60:46705)
W20260810 23:11:00.945087 6004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:11:00.945171 6004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:11:00.945135 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ad38a0c249473eb28de452d7fb3333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "249a6816b34044b3a83ead885db896a2"
I20260810 23:11:00.945384 5928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:11:00.945393 5967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.945560 5967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:11:00.946553 5992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:11:00.946633 5992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:11:00.946825 5928 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ad38a0c249473eb28de452d7fb3333 in term 1.
I20260810 23:11:00.947284 5752 leader_election.cc:302] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249a6816b34044b3a83ead885db896a2, 40ad38a0c249473eb28de452d7fb3333; no voters:
I20260810 23:11:00.948354 5852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40ad38a0c249473eb28de452d7fb3333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ac3d57c00114183b7f36f4f7694bb31"
I20260810 23:11:00.948559 5852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:11:00.949944 5852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ad38a0c249473eb28de452d7fb3333 in term 1.
I20260810 23:11:00.950476 5733 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260810 23:11:00.950609 5733 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:11:00.953529 5985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:11:00.953923 5985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [term 1 LEADER]: Becoming Leader. State: Replica: 40ad38a0c249473eb28de452d7fb3333, State: Running, Role: LEADER
I20260810 23:11:00.954021 5810 master_runner.cc:263] No leader master found from master 8ac3d57c00114183b7f36f4f7694bb31
I20260810 23:11:00.954087 5810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:11:00.954473 5985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } }
I20260810 23:11:00.956856 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ad38a0c249473eb28de452d7fb3333. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.957060 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: This master's current role is: LEADER
I20260810 23:11:00.959019 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:11:00.960038 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:11:00.965942 5928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [term 1 FOLLOWER]: Refusing update from remote peer 40ad38a0c249473eb28de452d7fb3333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:11:00.966851 6013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:11:00.971144 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ad38a0c249473eb28de452d7fb3333. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.971292 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:11:00.972664 5852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [term 1 FOLLOWER]: Refusing update from remote peer 40ad38a0c249473eb28de452d7fb3333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:11:00.973425 6013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:11:00.978763 5967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ad38a0c249473eb28de452d7fb3333. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.978952 5967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:11:00.987498 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.961934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.14.75.61:35889)
0810 23:11:00.962067 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.962072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.977742 (+ 15670us) server_negotiation.cc:407] Connection header received
0810 23:11:00.977903 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.977907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.977953 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.978000 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.979210 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.979952 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.981011 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.981365 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.981478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.983458 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.983462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.983465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.983495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.985305 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.985855 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.985859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.985861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.985957 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.986346 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.986351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.986354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.986623 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.986741 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.987160 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.987299 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:11:00.989338 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.952501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.14.75.60:46705)
0810 23:11:00.952634 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.952639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.974117 (+ 21478us) server_negotiation.cc:407] Connection header received
0810 23:11:00.974297 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.974300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.974348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.974399 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.977352 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.978106 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.980130 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.980486 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.980574 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.985241 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.985246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.985250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.985288 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.987246 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.987765 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.987770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.987774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.987866 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.988176 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.988183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.988186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.988446 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.988579 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.988935 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.989089 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:11:00.993294 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.993459 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: This master's current role is: LEADER
I20260810 23:11:00.993750 6016 mvcc.cc:205] Tried to move back new op lower bound from 7317108576107589632 to 7317108576073547776. Current Snapshot: MvccSnapshot[applied={T|T < 7317108576107589632}]
I20260810 23:11:00.993798 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:00.993918 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:11:00.996132 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:00.952852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.14.75.62:41041)
0810 23:11:00.953207 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:00.953212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:00.961776 (+ 8564us) server_negotiation.cc:407] Connection header received
0810 23:11:00.972609 (+ 10833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:00.972626 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:00.972687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:00.972750 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:00.975792 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.976527 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.981991 (+ 5464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:00.982369 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:00.982458 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:00.992043 (+ 9585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:11:00.992048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:11:00.992051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:11:00.992088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:11:00.994213 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.994727 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.994731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.994733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.994829 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:11:00.995159 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:11:00.995163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:11:00.995164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:11:00.995389 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:11:00.995499 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:00.995843 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:11:00.995970 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260810 23:11:00.996358 6018 mvcc.cc:205] Tried to move back new op lower bound from 7317108576107589632 to 7317108576073547776. Current Snapshot: MvccSnapshot[applied={T|T < 7317108576107589632}]
I20260810 23:11:00.999063 6015 catalog_manager.cc:1347] Generated new cluster ID: f6245751b7504e12bb22f158582839c4
I20260810 23:11:00.999140 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:11:01.000582 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:01.000699 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333 [sys.catalog]: This master's current role is: LEADER
I20260810 23:11:01.001884 5967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40ad38a0c249473eb28de452d7fb3333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ad38a0c249473eb28de452d7fb3333" member_type: VOTER last_known_addr { host: "127.14.75.62" port: 41041 } } peers { permanent_uuid: "8ac3d57c00114183b7f36f4f7694bb31" member_type: VOTER last_known_addr { host: "127.14.75.61" port: 35889 } } peers { permanent_uuid: "249a6816b34044b3a83ead885db896a2" member_type: VOTER last_known_addr { host: "127.14.75.60" port: 46705 } } }
I20260810 23:11:01.001996 5967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:11:01.004781 6020 mvcc.cc:205] Tried to move back new op lower bound from 7317108576107589632 to 7317108576073547776. Current Snapshot: MvccSnapshot[applied={T|T < 7317108576107589632}]
I20260810 23:11:01.028856 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:11:01.030315 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:11:01.042945 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ad38a0c249473eb28de452d7fb3333: Generated new TSK 0
I20260810 23:11:01.044016 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:11:01.079929 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.076346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.14.75.62:41041)
0810 23:11:01.076481 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.076485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.078522 (+ 2037us) server_negotiation.cc:407] Connection header received
0810 23:11:01.078957 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.078961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.079021 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.079091 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.079785 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36174: BlockingRecv error: recv got EOF from 127.0.0.1:36174 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:11:01.081102 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.077127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.14.75.60:46705)
0810 23:11:01.077239 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.077243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.078866 (+ 1623us) server_negotiation.cc:407] Connection header received
0810 23:11:01.080325 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.080329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.080379 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.080428 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.080971 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40812: BlockingRecv error: recv got EOF from 127.0.0.1:40812 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:11:01.082823 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.077900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.14.75.61:35889)
0810 23:11:01.078015 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.078019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.081784 (+ 3765us) server_negotiation.cc:407] Connection header received
0810 23:11:01.082029 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.082032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.082101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.082158 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.082694 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60540: BlockingRecv error: recv got EOF from 127.0.0.1:60540 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 23:11:01.087764 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.086064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.14.75.61:35889)
0810 23:11:01.086191 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.086195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.086686 (+ 491us) server_negotiation.cc:407] Connection header received
0810 23:11:01.086921 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.086924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.087003 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.087077 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.087654 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60556: BlockingRecv error: recv got EOF from 127.0.0.1:60556 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 23:11:01.093106 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.091457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.14.75.61:35889)
0810 23:11:01.091561 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.091565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.092115 (+ 550us) server_negotiation.cc:407] Connection header received
0810 23:11:01.092312 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.092316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.092376 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.092431 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.092965 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60566: BlockingRecv error: recv got EOF from 127.0.0.1:60566 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 23:11:01.099530 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.097692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.14.75.61:35889)
0810 23:11:01.097845 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.097850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.098546 (+ 696us) server_negotiation.cc:407] Connection header received
0810 23:11:01.098793 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.098796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.098843 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.098891 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.099423 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 23:11:01.107838 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.105971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.14.75.61:35889)
0810 23:11:01.106143 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.106147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.106743 (+ 596us) server_negotiation.cc:407] Connection header received
0810 23:11:01.106990 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.106994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.107063 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.107124 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.107719 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60582: BlockingRecv error: recv got EOF from 127.0.0.1:60582 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260810 23:11:01.120751 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.118715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.14.75.61:35889)
0810 23:11:01.118847 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.118852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.119598 (+ 746us) server_negotiation.cc:407] Connection header received
0810 23:11:01.119830 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.119834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.119892 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.119950 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.120632 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:11:01.141252 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.139445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.14.75.61:35889)
0810 23:11:01.139570 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.139574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.140191 (+ 617us) server_negotiation.cc:407] Connection header received
0810 23:11:01.140390 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.140394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.140446 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.140498 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.141114 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60596: BlockingRecv error: recv got EOF from 127.0.0.1:60596 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:11:01.177263 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.175862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.14.75.61:35889)
0810 23:11:01.176000 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.176005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.176289 (+ 284us) server_negotiation.cc:407] Connection header received
0810 23:11:01.176473 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.176477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.176533 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.176624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.177135 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60600: BlockingRecv error: recv got EOF from 127.0.0.1:60600 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 23:11:01.244828 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.243699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.14.75.61:35889)
0810 23:11:01.243850 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.243854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.244011 (+ 157us) server_negotiation.cc:407] Connection header received
0810 23:11:01.244151 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.244154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.244203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.244298 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.244719 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60614: BlockingRecv error: recv got EOF from 127.0.0.1:60614 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 23:11:01.376242 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.375149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.14.75.61:35889)
0810 23:11:01.375322 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.375325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.375449 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:11:01.375574 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.375576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.375625 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.375712 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.376140 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60616: BlockingRecv error: recv got EOF from 127.0.0.1:60616 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 23:11:01.635702 5884 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:11:01.634544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.14.75.61:35889)
0810 23:11:01.634699 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:01.634703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:01.634831 (+ 128us) server_negotiation.cc:407] Connection header received
0810 23:11:01.634936 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:01.634938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:01.634985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:01.635067 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:11:01.635558 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60622: BlockingRecv error: recv got EOF from 127.0.0.1:60622 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:11:01.942842 5991 catalog_manager.cc:1259] Loaded cluster ID: f6245751b7504e12bb22f158582839c4
I20260810 23:11:01.942934 5991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: loading cluster ID for follower catalog manager: success
I20260810 23:11:01.945575 5991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: acquiring CA information for follower catalog manager: success
I20260810 23:11:01.947423 5991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ac3d57c00114183b7f36f4f7694bb31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:11:01.948223 5992 catalog_manager.cc:1259] Loaded cluster ID: f6245751b7504e12bb22f158582839c4
I20260810 23:11:01.948313 5992 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: loading cluster ID for follower catalog manager: success
I20260810 23:11:01.950454 5992 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: acquiring CA information for follower catalog manager: success
I20260810 23:11:01.951594 5992 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 249a6816b34044b3a83ead885db896a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:11:02.153925 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:02.149920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.14.75.61:35889)
0810 23:11:02.150280 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:02.150283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:02.150372 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:11:02.150496 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:02.150498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:02.150635 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:02.150747 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:11:02.151007 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.151559 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.152385 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.152612 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.152693 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:02.153015 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:11:02.153585 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:11:02.153681 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:02.153755 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:11:02.153802 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":208,"threads_started":1}
I20260810 23:11:02.160645 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:02.157053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.14.75.60:46705)
0810 23:11:02.157272 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:02.157276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:02.157289 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:11:02.157385 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:02.157388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:02.157571 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:02.157682 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:11:02.157896 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.158500 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.159346 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.159573 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.159647 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:02.159901 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:11:02.160309 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:11:02.160399 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:02.160457 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 23:11:02.160506 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260810 23:11:02.167953 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:02.163807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.14.75.61:35889)
0810 23:11:02.164048 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:02.164053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:02.164280 (+ 227us) server_negotiation.cc:407] Connection header received
0810 23:11:02.164456 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:02.164461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:02.164512 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:02.164608 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:11:02.164918 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.165591 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.166775 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.167091 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.167191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:02.167470 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:11:02.167567 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:11:02.167643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:02.167752 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 23:11:02.167819 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260810 23:11:02.168114 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:02.164222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.14.75.60:46705)
0810 23:11:02.164344 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:02.164348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:02.164589 (+ 241us) server_negotiation.cc:407] Connection header received
0810 23:11:02.164727 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:02.164733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:02.164786 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:02.164830 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:11:02.165198 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.165967 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.166859 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.167175 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.167275 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:02.167593 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:11:02.167701 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:11:02.167768 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:02.167913 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 23:11:02.167975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:11:02.168782 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:11:02.163799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.14.75.62:41041)
0810 23:11:02.164065 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:11:02.164072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:11:02.164332 (+ 260us) server_negotiation.cc:407] Connection header received
0810 23:11:02.164442 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:11:02.164446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:11:02.164636 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:11:02.164759 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:11:02.165040 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.165890 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.167028 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:11:02.167358 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:11:02.167473 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:11:02.167903 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:11:02.168281 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:11:02.168363 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:11:02.168532 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:11:02.168597 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260810 23:11:02.171794 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5733
I20260810 23:11:02.180119 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5810
I20260810 23:11:02.186800 14636 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmgqrPD/build/debug/bin/kudu with pid 5886
2026-08-10T23:11:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3120 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6216 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290916 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST